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Introduction 


Despite the fact that education systems have been heavily investing in technol- 
ogy since the early 1980s, international indicators on technology uptake and use 
in education are missing. For more than 25 years education systems have been 
able to design and implement policies in this domain without those indicators, so 
the question is: why start discussing them now? Is the information available not 
good enough ? 


Why now? 

The existing international indicators still mirror the first policy priorities of the 
early 1980s: securing student access to computers and the Internet in schools. 
Indicators such as ratios of students per computer or percentage of schools with 
broadband access, although still a concern in some countries, do not yet provide 
the most relevant information for today’s policy in the field: how is technology 
used in schools? Is this use truly supporting the emergence of the learning envi- 
ronment that a knowledge-based society requires? 

Certainly, knowledge economies and societies would greatly benefit from a 
broader set of internationally comparable indicators. These could monitor 
progress in ICT uptake and unveil important information about use, ranging from 
issues such as frequency to purpose. If carried out in an international compara- 
ble framework they will become an important tool for benchmarking policies and 
practices across countries and over time. 

Our increasingly technology-rich world raises new concerns for education while 
also expecting schools to become the vanguard of knowledge societies. Firstly, 
technology can provide the necessary tools for improving the teaching and 
learning process, opening new opportunities and avenues. In particular, it could 
enhance the customisation of the educational process, adapting it to the particu- 
lar needs of the student. Secondly, education has the role of preparing students 
for adult life, and therefore it must provide students with those skills necessary 
to join a society where technology-related competencies are becoming increas- 
ingly indispensable. The development of these competencies, which are part of 
the set of the so-called ‘21st century competencies’, is increasingly becoming 
an integral part of the goals of compulsory education. Finally, in a knowledge 
economy driven by technology, people who do not master these competencies 
may suffer from a new form of digital divide that may affect their capacity to fully 
integrate the knowledge economy and society. 
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Because of these reasons, most countries have undertaken significant invest- 
ments to enhance the role of technology in education recently, after some years 
of less activity immediately after the implosion of the Internet bubble. Many 
would say that the incorporation of technology in education has lost its status 
as policy priority number one, although for a number of political reasons invest- 
ments have not been stopped. In many respects, the principle of ‘build it and 
they will come' seems to have taken root, and education systems keep investing 
in technology based on the belief that, sooner or later, schools and teachers will 
adopt it and benefit from it. The question that arises then is whether or not these 
new investments are paying off; is this investment in technology within educa- 
tion systems managing to fulfil expectations? 


New policy concerns, increased need for evidence 
and indicators 

Ironically, what countries have been investing in this field has hardly been the 
subject of any comparison. Therefore, countries can hardly claim that they are 
investing significantly in this. But even more important than the amount of effort 
invested, what really presses for an evidence-based policy debate about tech- 
nology in education is the emergence of new policy concerns. At least some of 
them, and the corresponding policy discussions, could benefit from more solid 
and comparable evidence: the emergence of a second digital divide, the need to 
promote the broad set of 21st century skills, and the still unfulfilled experience of 
promoting radical change in the provision of school education. 

First, recent evidence has unveiled that the digital divide in education goes 
beyond the issue of access to technology. A new second form of digital divide 
has been identified: the one existing between those who have the right compe- 
tences and skills to benefit from computer use, and those who do not. These 
competences and skills are closely linked to the economic, cultural and social 
capital of the student. This has important implications for policy and practice. 
Governments should make an effort to clearly convey the message that compu- 
ter use matters in the education of young people and they should do their best 
to engage teachers and schools in raising the frequency of computer use to a 
relevant level. Such an increase could not only be a clear indication of teachers’ 
and schools’ engagement with the development of 21st century skills and com- 
petencies, but it could also report gains in educational performance. In addition, 
schools should be reminded that they have a crucial role in the development of 
the cultural capital that will allow students to bridge the emerging second digital 
divide. 

Second, the changing needs of economic and social development require a wide 
range of new skills and competencies, known as the 21 st century competencies. 
These are considered key enablers of responsible citizenship in a knowledge- 
based and technology-pervaded economy. For instance, the recommendation 
of the European Parliament and the Council on key competences for lifelong 
learning defines a framework of eight competences considered important for 
the knowledge society. Digital competence is highlighted as one of the eight key 
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competences. In 2007 the Council identified a framework of 16 core indicators 
for monitoring progress in the field of education. ICT skills are a core indicator 
in this framework. Technology is hence expected to play an increasing role in 
education in the coming years. 

Last but not least, there is the pending issue of whether or not today’s teach- 
ing and learning experience in schools matches what could be expected from 
a knowledge society. The question is not which technology leads to increased 
productivity in education, but which new technology-supported methodologies 
improve student performance over traditional ones, if any at all, and which other 
factors intervene. Previous calls have already been made in order to investi- 
gate the explicit relationships among technology, instructional strategy, psycho- 
logical processes and contextual factors. The almost infinite array of methodo- 
logical possibilities makes this kind of investigation extremely difficult, but not 
impossible, provided that there is sufficient effort devoted to the accumulation 
and dissemination of the resulting knowledge base. Such a task might appear 
overwhelming, particularly as the technological frontier is constantly changing. 
However, it is worth the effort. And policymakers and researchers cannot be in a 
position to monitor what is truly going on in schools unless critical indicators about 
intensity, purpose and context of use of technology in education are available. 

A truly international effort 

Therefore it is relevant to assess and compare how education systems are deal- 
ing with technology integration in schools — particularly in terms of securing 
and improving access, enhancing a wide range of educational and managerial 
uses, and monitoring the effects and impacts on the development of critical 
technology-related skills and competencies. Such a comparison is not possible 
in the absence of appropriate indicators which, at the moment, are missing in 
the international collections already available. 

Both the European Commission and OECD have recognised the need for reli- 
able indicators in the area of technology in education. OECD has raised this 
issue in the context of the recently published report Beyond Textbooks. Digital 
Learning Resources in the Nordic Countries. It highlights the need for a compre- 
hensive approach to indicators on technology in education and the difficulties 
associated with their development and data collection. The same need has also 
emerged during the analysis of the relationship between technology use and 
educational performance drawing on PISA 2006 data, which will be published by 
CERI in 2009. The European Commission has initiated several studies intended 
to summarise existing and available information in the field. 

Other international organisations, such as Unesco, the World Bank and the Inter- 
American Development Bank, share similar needs and are willing to cooperate 
in this process. An inter-agency seminar carried out in Korea in July 2009 ('), 
provided an excellent opportunity to compare priorities and agree on the need 
to explore further synergies. 


(’) see http://go.worldbank.org/DJTDITWI40 
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What this volume adds to the discussion 

It is within this context that the present volume has to be understood. The con- 
tributions included stem from an international expert meeting which took place 
in April 2009 when the Centre for Research on Lifelong Learning (CRELL), in 
cooperation with CERI, organised and hosted an international expert meeting 
on the issue of benchmarking technology use and effects in education. The 
workshop specifically aimed at constructing a framework to look at the relevant 
domains and interdependence between components related to ICT in educa- 
tional processes from a holistic perspective ( 2 ). 

This book is organised into four different sections. The first one looks into the 
context of ICT impact assessment in education. This chapter addresses the 
political context and includes reflections about the assessment needs at an 
international level. 0ystein Johannessen follows a policy perspective. He dis- 
cusses the challenge of developing benchmarks and the need to incorporate 
a multi-faceted approach which takes into account the complexity of issues to 
consider when setting up a knowledge-base on ICT in education. In his article, 
Ola Erstad maintains the need for a broadened understanding by policymakers 
of impact and outcomes. Based on experience gained in Norway, he suggests 
a multilevel approach and tries to identify key indicators of impact for all the dif- 
ferent levels addressed. 

The second chapter is about the state of the art of ICT impact assessment. A 
conceptual overview on educational monitors is provided by Willem Pelgrum, 
who introduces the various dimensions and challenges of ICT assessment and 
methodologies issues in international comparative monitoring. Michael Trucano 
then presents conclusions from the World Bank series of ‘knowledge maps' about 
ICT in education. Despite a variety of useful resources, he identifies important 
gaps and a lack of reliable impact evidence in order to better support the effec- 
tive integration of ICT in developing countries. 

Conceptual frameworks are discussed in Chapter 3 in order to agree on a gen- 
eral common understanding about aspects to take into account when assessing 
the effects and impact of ICT in education especially for comparative purposes 
at country level. A conceptual framework should provide an orientation for any 
kind of measurement required in the decision-making process and act as a ref- 
erence which is flexible and adaptable to specific purposes of studies to be car- 
ried out. It should also provide a holistic view and support the setting of standard 
orientations when defining the evaluation methodology and selecting appropri- 
ate instruments for measurement. The framework developed by Katherina Kikis, 
Friedrich Scheuermann and Ernesto Villalba for the Joint Research Centre of 
the European Commission aims to contribute to a systematic approach on how 
to identify the use of ICT and its effects on all different levels and stages. A simi- 
lar approach is then presented from Marcelo Cabrol and Eugenio Severin, which 
is currently being implemented in projects of the Inter-American Development 
Bank in Latin America and the Caribbean. Finally, Befiat Bilbao and Francesc 


( 2 ) Contributions are published at the CRELL website (http://crell.jro.it/workshopimpact.htm). 
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Pedro discuss the conceptual approach proposed by the OECD for looking into 
the impact of digital learning resources and benchmarking the use in school 
education. 

A series of reflective case studies are presented in Chapter 4. One important 
aspect of ICT impact assessment is to be clear on what is to be assessed at 
the individual level and to think about appropriate ways of measurements. 
Technology use and critical thinking and problem-solving approaches (‘new 
literacies') are discussed by Edys S. Quellmalz in the context of assessment 
design and implementation. She looks at current approaches in assessment 
and underlines the need to reach consensus about what is to be measured. ICT 
implementation policies in education in Hong Kong are then analysed by Nancy 
Law, Yeung Lee and H. K. Yuen in terms of their impact on teaching and learn- 
ing processes. They also present an interesting research design and concepts 
of information literacy assessment. Willem Pelgrum then reports about moni- 
toring scenarios and sets of indicators on the use and impact of ICT in primary 
and secondary education. His work is based on the results of a study carried 
out in the European Union which can be seen as a further step to implement 
mechanisms for regular ICT implementation monitoring at a European level. A 
theoretical framework of various factors affecting ICT use in education is pre- 
sented by Heeok Heo and Myunghee Kang. This framework had been embed- 
ded in a nationwide investigation in Korea. Findings clearly indicate that a better 
understanding of the real impact can only be achieved if more consideration is 
given to the use of ICT in informal learning. In addition, results from a compara- 
tive analysis in the European countries on ICT in primary education are then 
described by Roger Blamire. The approach was based on an analytical frame- 
work allowing an examination of the impact on three different levels: on learning 
and learners, on teachers and teaching and on primary school development 
plans and strategies. Altogether these cases help to better understand the need 
for comprehensive studies of the complex interactions between various types of 
ICT implementation and its effects, including other factors to take into account 
which have not yet been addressed by existing studies. 

The aim of this book is to provide a basis for the design of frameworks, the 
identification of indicators and existing data sources as well as gaps in areas 
where further research is to be initiated. The contributions clearly demonstrate 
that there is a need for the development of consensus around widely accepted 
approaches, indicators and methodologies. In this context more harmonisation 
of existing survey approaches would be desirable. Therefore, this collection of 
articles follow the intention of both organisations, the OECD and the European 
Commission, to foster international cooperation with other relevant international 
organisations and to serve as a starting point for common reflection on ways to 
assess how ICT is used in education. 

Without such an assessment, it is virtually impossible to make any progress 
in the direction of understanding better how the actual pedagogies are trans- 
formed and which policies, both at national and local levels, are making a dif- 
ference. Only a truly international comparative effort can provide the necessary 
evidence. And even if the contributions in this book show a vast diversity of 
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perspectives, at least they point in the right direction. Even more important than 
getting the hard evidence is to make significant progress in understanding the 
worth of technology in education and in how to measure progress. This book has 
to be seen as a serious attempt to touch base and, as such, has to be taken as 
the beginning of a journey. The sooner we start walking the better. 


Friedrich Scheuermann 


Francesc Pedro 
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In search of the sustainable knowledge 
base: multi-channel and multi-method? 

0ystein Johannessen 

Ministry of Education and Research, Norway 


Abstract 

This article discusses the need for developing an open, flexible and international 
knowledge base for ICT in education, in which joint development of benchmarks 
can play a key rote for addressing complexity, multi-stakeholder interests and 
international comparisons. The need for a multi-channel and multi-method approach 
is elaborated. The article is written from the point of view of a policymaker. 


Introduction 

ICT (information and communication 
technologies) in education lives a life 
at the crossroads between evidence- 
based policymaking, learning and the 
fast-changing world of technology. 
Key stakeholders (politicians, parents, 
teachers, school leaders) demand 
evidence of the impact of ICT derived 
from research, monitoring and evalu- 
ation. The challenge for policymakers 
is (in collaboration with the research 
community and the educational com- 
munity) to develop a sustainable 
knowledge base for ICT in educa- 
tion, in which key indicators and other 
sources of information are identified, 
which enables better insight into the 
use and effects of ICT for learning. I 
have chosen to discuss the issue of 
developing benchmarks for ICT in 
education, because benchmarks are 
embedded in the evolving knowledge 
base in this field. 

This article is structured in four parts. 
In the first part, I describe the policy 
backdrop, within which the issue of 


developing a sustainable knowledge 
base should be discussed. The second 
part focuses on the issue of what we 
have learnt from R & D with regard 
to the effects of ICT in education. In 
the third part, I describe the concept 
of the multi-channel and multi-method 
knowledge base, before I finish with 
some remarks on the issue of a sys- 
temic approach to benchmarks and 
other critical components of a knowl- 
edge base for ICT in education. 

This article is written from the point of 
view of a policymaker. 

What is the rationale 
behind the focus on ICT in 
education? 

ICT in education has, in recent years, 
emerged as a policy area. Many coun- 
tries have developed ICT strategies, 
either as separate strategies or as 
strands embedded in national strate- 
gies for education or for the devel- 
opment of the information society at 
large in the country. The strategies 
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and their underlying rationales share 
many common features. 

Kozma (2008) has identified impor- 
tant reasons for investing in ICT for 
education. 

To support economic growth mainly 
by developing human capital and 
increasing the productivity of the 
workforce. 

To promote social development by 
sharing knowledge, fostering cultural 
creativity, increasing democratic 
participation, improving access to 
government services and enhancing 
social cohesion. 

To advance education reform, i.e. 
major curriculum revisions, shifts in 
pedagogy or assessment changes. 
To support educational management 
and accountability, with an empha- 
sis on computer-based testing and 
the use of digital data and manage- 
ment systems. 

These features relate the issue of ICT 
in education to its function in a broader, 
societal context. The role of ICT in 
education must also be linked to edu- 
cational needs. In many countries, the 
role of ICT is linked to issues of educa- 
tional attainment and the importance 
of ICT for advancing robust learning 
strategies on the side of the students. 
A second area is ICT as a tool for the 
support of personalisation strategies in 
teaching and learning. ICT can also be 
used to increase visualisation and var- 
iation in many subjects. As a greater 
proportion of our homes are linked to 
the Internet, the role of ICT in home/ 
school access is now being exploited. 
Many children start to use ICT at an 
early age, and the home and the family 
are, in many cases, an arena for the 
initial acquisition of digital skills. Thus, 
education has a role to play in further- 
ing these skills, based on pedagogical 


principles. Our educational systems 
should bear in mind that ICT should be 
an integral part of learning, in order to 
provide learners from families with a 
low socio-economic status with neces- 
sary digital skills for learning, work and 
life in order to avoid digital divides. 

ICT is not integrated in education for 
its own sake. A proper integration of 
ICT in key policy priorities in differ- 
ent countries can be a productive 
approach in order to secure ICT as 
a mainstream part of education. In 
Norway, ICT is not subject to a sepa- 
rate strategy; it is rather embedded 
in the national curriculum and linked 
to overall political priorities stated by 
the government: quality of learning, 
higher completion rates and students’ 
well-being and mastery. 

What have we learnt from 
R&D? 

We have been through a period in 
which politicians and policymakers 
have focused on the need for estab- 
lishing credible proof for the return on 
investments in ICT. This has resulted 
in a search for causal relationships 
between ICT and educational quality, 
i.e. learning outcomes. As the OECD 
(2008) has pointed out, this has been 
difficult to achieve because of the 
lack of large-scale, longitudinal stud- 
ies and a lack of methodologies that 
can capture the complexity of ICT and 
other elements influencing educational 
quality. 

One of the most significant studies to 
date is the lmpaCT2 report from 2002 
(Harrison et al., 2002). The study 
shows that ICT leads to statistically 
significant improvements in some sub- 
jects, whereas there are no significant 
improvements in other studies. The 
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OECD, through its work on the PISA 
studies, has been able to demonstrate 
interesting correlations between home 
access and use of ICT on the one hand 
and PISA score on the other hand. The 
relation between ICT use at school 
and PISA score is fare more complex. 
So far, these correlations have not 
been explained. The study ‘E-learning 
Nordic’ (Ramboll Management, 2006), 
which looks at the perceived impact of 
ICT, shows that all stakeholders (stu- 
dents, parents, teachers, principals) 
believe that ICT can have a positive 
impact on teaching and learning. 

The studies and reports mentioned 
above represent a plethora of studies. 
The European Schoolnet shows in its 
metastudy on impact studies (EUN, 
2006) that there are a number of stud- 
ies, also related to patterns of use 
across the technological spectrum. 
Impact studies cover a wide spectrum 
between the search for causal relation- 
ships between ICT and educational 
attainment on the one hand and stud- 
ies looking at the perceived impact of 
ICT on the other hand. 

The focus of some studies has been 
on causality and on quantitative issues 
regarding ICT use. It is time to review 
critically whether we have been asking 
the right research questions. In its first 
report on ICT and PISA score (OECD, 
2004), the OECD states: 

‘It is the quality of ICT usage, rather than 
necessarily the quantity, that will deter- 
mine the contribution that these technolo- 
gies make to students’ outcome.’ 

Instead of looking for causality, we 
need to ask how we can improve and 
optimise the use of ICT in teaching and 
learning, and in doing so we also need 
to listen to the voices of the learners 
and the practitioners. 


Multi-channel: 

ICT in education covers 
a wide spectrum 

The first pillar of my approach to a sus- 
tainable knowledge base is the reali- 
sation that ICT in education covers a 
wide spectrum — both thematically 
and along the administration-peda- 
gogy axis. This is a consequence of 
the incremental integration of ICT into 
all domains of education. 

Kozma (2008) has highlighted this in 
his work, and he acknowledges that 
ICT strategies in many countries cut 
across diverse fields. 

• Infrastructure development is nec- 
essary in order to ensure access to 
schools, networks and resources for 
learning. 

■ Teacher training, both initial and in- 
service, is a prerequisite for the abil- 
ity of education to use ICT in learn- 
ing processes. 

Technical assistance is needed both 
in the administrative as well as in 
the pedagogical domain. 

Curricula and pedagogical 
approaches may have to be changed 
in order to cater for educational 
change with ICT. 

Content development is necessary 
in order to facilitate the interactive 
potential ICT can offer in the teach- 
ing and learning process. 

In my opinion, a multi-channel 
approach to the knowledge base is 
necessary in order to be able to ask 
the right questions and to grasp the 
plethora of issues related to ICT in 
education. Let me elaborate on a few 
issues. 

■ It is necessary to continue the moni- 
toring of infrastructure development. 
Although many countries have 
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developed a superb infrastructure, 
access to ICT is still an issue in many 
European countries. This is truly the 
case if you look at access issues on 
a global scale. The same goes for 
the need for monitoring the evolving 
patterns of use. We need to be able 
to assess the speed of uptake of dif- 
ferent technologies for learning as 
well as assessing the degree of vari- 
ation across the spectrum of learn- 
ing technologies. A particular chal- 
lenge with regard to monitoring the 
patterns of use is the high degree of 
technological and cultural diversity 
that is to be expected in many coun- 
tries around the globe. 

Gender issues are visible. PISA 
data show that although the gap 
between genders is closing, there 
are still interesting differences to 
be found with regard to patterns of 
use. A fairly new dimension regard- 
ing gender issues is that it might 
be just as important to study differ- 


ences within a gender as between 
genders. 

Digital learning resources (DLR) 
are characterised by complexity — 
a crossroads between pedagogy, 
technology, IPR and the market- 
place. This is an area which, in my 
opinion, has been under-assessed, 
and we need a stronger focus both 
on benchmarking of digital learn- 
ing resources as well as a research 
agenda for DLR and learning. 

For PISA (2003) and PISA (2006), 
follow-up analysis based on ICT data 
has been undertaken. In future, the 
ICT analysis of PISA should be rep- 
licated and improved, and the ICT 
familiarity questionnaire should be 
updated in order to keep up with the 
evolving use of ICTs for learning. 
Few countries have developed good 
methodologies for assessing digital 
skills among students. Such meth- 
odologies should be developed both 
within and across subjects. 
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Figure 1: Pupil use of digital content, computer games, mobile phones and office programs — seventh grade, 
ninth grade and VK1, where 'daily' and 'weekly' have been merged (in percentages). 
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Some countries are monitoring both 
access and use of ICT. The Norwegian 
ITU Monitor (Arnseth et al., 2007) is 
a biannual monitor that assesses the 
status with regard to ICT in Norwegian 
schools. The following figure shows 
an example of patterns of use among 
Norwegian students. 

The list of topics shows that there are 
many phenomena in ICT and learn- 
ing that should be monitored and 
assessed through a variety of chan- 
nels, but is this enough? In the next 
chapter I will elaborate on the need for 
a multi-method approach in order to 
ensure a sustainable and systemically 
coherent knowledge base. 

Multi-method approach 
to the knowledge base 

A consequence of the increased focus 
on evidence-based policymaking is 
that national authorities need to move 
away from anecdotal and unsystem- 
atic evidence of how ICT is being 
used in education and how it impacts 
teaching and learning. Such a change 
of focus highlights requirements of 
methodology and validity. The multi- 
tude of issues at hand, which I have 
described in the preceding chapter, 
and the need for diverse approaches 
indicates that building a sustainable 
and flexible knowledge base requires 
a combination of quantitative and 
qualitative methods. Furthermore, a 
system of indicators and other input to 
the knowledge base must be flexible 
enough to allow for changing patterns 
of use and the emergence of new 
technologies for learning. 

An important question is whether the 
methods are good enough, and ifthere 
is room for improvement. Awell-known 
challenge in educational research and 


development is to be able to capture 
the complexity of the learning proc- 
ess. In my view, we need to further 
explore the potential of ethnographic 
research and so-called test-bed stud- 
ies. However, a downside to these 
approaches is that they are consum- 
ing both in terms of time and money. 

In the last couple of years, we have 
seen projects in several countries 
aiming at capturing the voices of the 
learners. One example of this is the 
digital generation project, funded 
by the MacArthur Foundation pro- 
gramme for digital media and learn- 
ing. The project conveys how children 
develop engagement, self-directed 
learning, creativity and empowerment 
through the use of digital media. Our 
educational systems need to develop 
our ability to listen to and reflect on 
the voices of the learners in order to 
understand how digital media influ- 
ence the lives and learning of our chil- 
dren. This topic will be addressed in 
the second half of the OECD new mil- 
lennium learners’ project. 

Digital media play a much bigger 
role in the lives of our students today 
than before. A Norwegian report from 
2008 (Arnseth et al., 2008) shows that 
more than nine out of 10 adolescents 
aged 16 to 19 use social media, and 
three out of four use social media on 
a daily basis. This raises the question 
of whether only ICT use in schools 
should form the basis of our under- 
standing of digital media and learning. 
We may have to broaden the scope 
and include out-of-school use of dig- 
ital media, given the extensive home 
use of digital media. This would also 
acknowledge the fact that the home 
of youngsters is the first arena for the 
acquisition of digital skills, albeit an 
informal, but nevertheless important 
arena. 
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A system of benchmarks 

As I wrote in the introduction to this 
paper, benchmarking is an integral 
part of the knowledge base national 
authorities, and the research and edu- 
cational community must develop. 
Developing a system of benchmarks is 
an exercise that requires careful plan- 
ning and solid reflections on the selec- 
tion and usability of benchmarks. 

As a point of departure for discussion, 
it is possible to distinguish between 
different types of benchmarks for ICT 
in education. I have divided them 
into first, second and third order 
benchmarks. 

First order benchmarks are typically 
related to access to ICT. This could 
be pupil: PC ratio and broadband 
access. 

Second order benchmarks try to 
capture in what ways and to what 
extent ICT is used in teaching and 
learning. These benchmarks can 
cover a wide range of use patterns 
and learning technologies, and they 
should capture both teachers’ and 
students’ use of ICT for learning. 
Third order benchmarks should 
cover the impact of ICT in teaching 
and learning. Benchmarks should 
be related to learning outcomes and 
learning strategies. 

Development of benchmarks should 
pay attention to the need for research 
and development in order to meet 
demands for validity and methodo- 
logical rigour. Many countries have 
elaborated benchmarks of the first and 
second order, but it has proved diffi- 
cult to develop solid third order bench- 
marks. Further research efforts should 
therefore be directed at the develop- 
ment of such benchmarks. 


Another important consideration 
regarding the benchmarking of ICT 
in education is related to the search 
for precision and validity. Given the 
complexity of education, underlying 
research-based concepts and models 
will inevitably reach a high level of 
sophistication. Flerein lies a danger. 
The models can be too ambitious in 
their strive for perfection, and it is 
important to realise that the concepts 
and models behind benchmarks must 
find an equilibrium between simplicity 
and complexity, because, by the end 
of the day, they should meet the needs 
of policymakers and practitioners. 

Systemic challenges 
related to development 
of benchmarks 

The development of benchmarks does 
not happen in a vacuum; it serves 
purposes related to decision-making, 
informed choices and the need for a 
deeper understanding of ICT in edu- 
cation and its development. It is, how- 
ever, difficult to know with great preci- 
sion what we are looking for, because 
ICT is embedded in pedagogical 
practice. This is especially pertinent 
for the development of third order 
benchmarks. 

Another systemic challenge is related 
to the trend in recent years that educa- 
tion has evolved into an arena for solv- 
ing many problems society as a whole 
and younger cohorts in particular are 
facing. The educational community 
may at times feel that it is under siege. 
Thus, development and utilisation of 
benchmarks that represent an admin- 
istrative burden should be carried out 
with great caution. 

A particular advantage related to 
benchmarks is that they are well 
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suited for international comparisons. 
However, so far little work has been 
done to develop an agreed interna- 
tional framework for benchmarking 
ICT in education. It should be in every- 
one’s interest to develop an interna- 
tional benchmarking framework. This 
could be done in a joint OECD-EU 
collaboration. One important consid- 
eration is to agree on common topics 
for benchmarking, and it is in my opin- 
ion vital to make sure that a sufficient 
spectrum of issues is addressed. 
Digital learning resources are a good 
case for benchmarking development, 
because DLR has a high degree of 
complexity, they are important for the 
quality of learning and there is too little 
evidence on the impact of DLRs. 

Developing a framework for bench- 
marking is a challenge that cannot be 
solved by one party alone. It is vital to 
ensure that such a framework should 
be developed in a triangular collabora- 
tion between researchers, policymak- 
ers and practitioners. The notion of 
‘methodological validity’ is important 
in research and benchmarking. When 
it comes to benchmarking of ICT and 
the issue of power of definition of what 
we are looking for to benchmark, it is 
in my opinion interesting to combine 
methodological validity with the notion 


of political validity. By political valid- 
ity I mean (in the context of discuss- 
ing benchmark development) that the 
choice of benchmarks should not only 
be directed by methodological per- 
spectives, it should also pay attention 
to the needs of key stakeholders in 
education when it comes to the choices 
of benchmarks. As such, developing 
benchmarks should take place at the 
crossroads between policy, practice 
and research. Methodological validity 
ensures that we can trust the informa- 
tion we get from benchmarks, political 
validity ensures that stakeholders in 
politics and society get the information 
they need. 

The road ahead? 

Benchmarking can play a role in 
developing an open knowledge base 
for ICT in education. International 
collaboration is necessary for such a 
venture because of complex issues, 
a wide spectrum of stakeholders and 
the need for agreed frameworks for 
international comparisons. By the end 
of the day, the knowledge base should 
be there to guide us in informed 
choices for the benefit of current and 
future cohorts of learners. Because 
they deserve it! 
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Abstract 

Within research on ICT and school development there is an increased understand- 
ing of the complexity involved in such processes. However, the focus on indicators 
and the impact of ICT in education from a policy perspective have been oriented 
towards a more narrow understanding of impact and outcomes, especially on the 
individual level. This article argues for the need for a multilevel approach towards 
ICT in education in order to fully understand the impact of such technologies in 
the education system. In the first part, some theoretical reflections on change and 
the research on impact are presented. In the second part, some examples will be 
described, mostly from a Norwegian setting, and in the last part, some key indica- 
tors of impact on different levels will be discussed. 


Introduction 

The most important point I have 
learned by studying the impact of ICT 
(information and communication tech- 
nologies) on Norwegian education 
during the last 10 years is the com- 
plexity and multilevel aspects of such 
innovations. The challenge is not so 
much to develop indicators for ICT in 
education as such. At present there 
are several available frameworks of 
indicators, about implementation of 
ICT in educational settings, about dig- 
ital literacy, about leadership and so 
forth. The challenge is rather to study 
ifferent levels and domains at the 
same time, and to bring different sets 
of indicators together into one strategy 
in order to assess the broad scope of 
impact of ICT on education. 


In recent years, there has been a ten- 
dency to argue that complexity is an 
issue in itself in studying knowledge 
practices (Law and Mol, 2002) or stud- 
ies on ICT, development and schools 
(Engestrem, Engestrem and Suntio, 
2002; Thomson, 2007). In order to 
fully understand or assess the effects 
of ICT in education we need to know 
more about how ICT operates on dif- 
ferent levels, and what we are really 
measuring on which levels. It is crucial 
that we synthesise the research with 
a holistic perspective in order to lay a 
foundation for further development in 
this area (Sutherland, Robertson and 
John, 2009). In this article, the argu- 
ment is built around the need to look 
at the bigger picture in order to create 
sustainable developments throughout 
our education systems, and under- 
stand ICT as a catalyst for change 
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on different levels. This creates 
challenges for the development of 
indicators of the impact of ICT in edu- 
cation since several sets of indicators 
need to be developed and different 
methods must be used. The objective 
would be to build a model that looks 
at how different levels and dimensions 
work together to create conditions for 
change and the integration of ICT in 
educational practice. 

Understanding change 

A major challenge for developments 
within technology and education today 
is to grasp the complexity of such devel- 
opments. In general, there has been 
a tendency to simplify the research 
approaches and understanding of 
how digital technologies might have 
an impact on schools and educational 
outcomes (Cuban, 1986, 2001; Erstad, 
2004), and evidence of the impact of 
ICT on educational practice has mainly 
been drawn from small-scale case 
studies (Condie and Munro, 2007). 
Both policymakers and researchers 
have created expectations towards the 
impact of information and communica- 
tion technologies on student learning, 
which has not gained strong support 
in the research literature (ibid.). Much 
research has been oriented towards 
the new possibilities and limitations 
created by the implementation of digital 
technologies into educational settings 
(De Corte, Verschaffel, Entwistle and 
van Merrienboer, 2003). Again, other 
research and development initiatives 
have been more directed towards the 
institutional framework of school devel- 
opment and the use of ICT (Krumsvik, 
2009). In later years, there has also 
been a growing interest for networks, 
both online and offline (Veugelers and 
O’Hair, 2005). The argument goes 
that digital technologies have created 
a new situation for how organisations 


and people work together and relate 
to each other, as a globalising proc- 
ess (Castells, 1996). Education is also 
thought of in a more distributed way 
by using these technologies for edu- 
cational purposes, such as in compu- 
ter supported collaborative learning 
(CSCL). 

The challenge, and the complex- 
ity, rests on how these levels and 
perspectives relate to each other. 
This is a challenge of educational 
research in general, but especially 
when trying to understand the mech- 
anisms involved in the educational 
use of ICT. In the research literature 
there is now a greater consciousness 
towards multilevel analysis (Van Dijk, 
2009) and more holistic approaches 
towards learning and school devel- 
opment (Hakkarainen, Palonen, 
Paavola and Lehtinen, 2004; Arnseth 
and Ludvigsen, 2006; Sutherland, 
Robertson and John, 2009). As David 
Olson has pointed out in his book 
Psychological theory and educational 
reform (2003): 

The problem, I believe, is that the theo- 
ries that gave us insight into children’s 
understanding, motivation, learning and 
thinking have never come to terms with 
schooling as an institutional practice with 
its duties and responsibilities for basic 
skills, disciplinary knowledge, grades, 
standards, and credentials... What is 
required, then, is an advance in our under- 
standing of schools as bureaucratic insti- 
tutions that corresponds to the advances 
in our understanding of the development 
of the mind. (D. Olson, 2003:x-xi) 

Olson argues that the challenge is to 
combine different levels in our under- 
standing and analysis of key charac- 
teristics of how schools function as 
learning organisations, and also the 
conditions for changes of activity at 
different levels. 
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In his classic book The new meaning 
of educational change (1 991), Michael 
Fullan presents a broad framework on 
different levels and involving different 
actors in understanding educational 
reform and school development. Also 
in his later book Change forces (1 993), 
he addresses the real complexity 
of dynamic and continuous change, 
showing the challenges this implies 
both for peoples’ mind-sets and for 
mechanisms defining educational 
practices. This has made the research 
community understand that change 
was not an event that occurred in such 
a way that a ‘before’ and 'after' could 
be recognised and measured; rather, 
he defined change as a process. 

In recent years, this has been taken up 
by other researchers trying to develop 
models to study and also to create 
interventions into educational prac- 
tices in order to work towards school 
development. This represents a move- 
ment away from traditional models 
of change based on organisational 
theory such as Senge or Nonaka and 
Takeuchi, towards models trying to 
grasp the complexity of change proc- 
esses through the activities involved. 
The most important perspective 
for studying change processes in 
schools in recent years has been 
activity theory, or more specifically 
cultural-historic activity theory (CHAT) 
(Engestrom, 1987). This has grown 
out of the intellectual work done by 
the Russian psychologist Vygotsky, in 
the 1920s and 1930s, and later on by 
Leontjev. The focus of this perspec- 
tive is on activity as the unit of analy- 
sis and mediation between actors and 
certain cultural tools. Yrjo Engestrom 
has then expanded this model beyond 
the person and the tools by introduc- 
ing a larger framework of factors that 
are part of developmental processes 
on different levels, such as rules and 


norms, division of labour and commu- 
nities of practice. The relation between 
these factors is defined as an activity 
system, and within an organisation and 
between organisations there might be 
several activity systems that relate to 
each other in different ways. 

The complexities of knowledge crea- 
tion and knowledge building have 
been an issue within research com- 
munities dealing with CSCL, studying 
how collaborative and distributed ways 
of working using different technologi- 
cal applications stimulate knowledge 
building among learners. This can be 
seen in the developmental work done 
by Marlene Scardamalia and Carl 
Bereiter in Canada (Scardamalia and 
Bereiter, 2006). Knowledge building, 
and the technological platform that has 
been developed (Knowledge Forum), 
aim for collective cognitive responsibil- 
ity among learners. Collective respon- 
sibility refers to a condition in which 
responsibility for the success of the 
group is distributed across all mem- 
bers rather than being concentrated on 
the leader. Collective cognitive respon- 
sibility refers to taking responsibility to 
know what needs to be known on the 
cognitive level in addition to the more 
tangible practical aspects. 

Networking is a broad conceptualisa- 
tion based on global perspectives on 
social development, but which also 
relates specifically to the role of edu- 
cation in moving towards knowledge 
societies and the role of networking 
in such processes. As an example, in 
the Unesco report Towards knowledge 
societies (2005), the concept of learn- 
ing is closely tied to innovation and 
networking. Crede and Mansell (1998) 
have also shown how this thinking on 
knowledge societies and networking 
is fundamentally based on identifying 
new ICT opportunities. 
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In his literature review on ‘whole school 
change’, Thomson emphasises that: 
The ways in which we think about the 
school also impact on what counts 
as change. There are two important 
aspects to thinking about change in 
schools: (1) understanding the school 
as an organisation, and (2) under- 
standing that change will be multi- 
layered’ (2007:15). In his presentation 
of a framework for change, he focuses 
on two important themes: ‘the timing 
of, and time for, whole school change’, 
and ‘a supportive framework’. 

The impact of ICT has become a key 
factor in many studies in understand- 
ing how new technologies both might 
be a catalyst and a driving force for 
change processes in themselves, and 
also an element that supports change 
within organisational settings. All this 
points towards a stronger emphasis 
on multilevel approaches studying 
change and the impact of ICT on dif- 
ferent levels within the same analysis. 

Systemic impact, 
curriculum developments 
and future competencies 

Such broader conceptions of change 
are important in developing an under- 
standing of key factors influencing 
educational practices (see examples 
below). The last decade has been 
associated with an upscaling of activi- 
ties using ICT in educational settings. 
From small groups of students and 
teachers, we have seen a rise in the 
way ICT has been implemented across 
the curriculum. The consequences in 
many countries have been that whole 
school communities use such tech- 
nologies in different activities, and that 
these developments have an impact 
on a national level through curriculum 
developments. 


An upscaling of activities has brought 
about a need for development of indi- 
cators that capture the more systemic 
developments of ICT in education, 
and how that transcends to the micro 
level of teaching and learning by teach- 
ers and students: not how we change 
single schools in the way they work 
with ICT, but rather how all schools and 
the school system as such experience 
changes by implementing and using 
ICT. 

One example is the national cur- 
riculum in Norway, from 2006, which 
defines ‘the ability to use digital tools’ 
and digital competence as a basic 
skill throughout the curriculum. In 
this way, the Ministry of Education 
and Research has placed a strong 
emphasis on ICT as part of learning 
activities in schools. ICT should be an 
integrated part of learning activities 
among all students, at all levels of pri- 
mary and secondary education and in 
all subjects. This also challenges how 
schools are organised. 

The focus on ICT and digital compe- 
tence in the new national curriculum 
builds on former plans and docu- 
ments. At the same time, it points 
towards future competencies, what 
are also termed as 21st century skills 
(www.21stcenturyskills.org). The impor- 
tant implication for the discussions 
in this article is the commitment this 
implies for teachers and students to 
use ICT much more broadly in the 
learning activities in schools. In this 
way, a stronger push mechanism is 
created for school leaders and teach- 
ers to work towards capacity building 
on school development and the use of 
ICT in order to fulfil the challenges of 
the new curriculum. Important national 
objectives related to the new national 
curriculum can be summarised as 
follows: 
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■ a focus on how ICT can contribute to 
an increased quality in teaching and 
learning; 

• an increased use of new ICT-based 
means for cooperation and inter- 
change of knowledge and experi- 
ence at all levels of the educational 
system; 

• a broad access to learning materi- 
als and the development of new and 
varied forms of learning in order to 
stimulate activity, independence and 
cooperation; 

• an increased focus on students’ crit- 
ical reflection with respect to the use 
of ICT in teaching and learning and 
in society in general; 

• an increased focus on how to avoid 
creating digital divides. 

Such curriculum developments also 
point to the need for multilevel analy- 
sis of the ways we study the impact 
of ICT on education. And it brings up 
a future observation of and an orien- 
tation towards the competencies stu- 
dents need today and in the future and 
that our school system needs to take 
into consideration. Competencies are 
here understood on different levels, 
not only seeing competency as an 
individual ability, but also on the col- 
lective level and the school level. In 
a Norwegian context, we have had 
different projects and strategies in 
developing indicators on these differ- 
ent levels, also trying to define what 
is called ‘the digital competent school’ 
or ‘digital maturity’. 

Before I move on to some reflections 
on a multilevel approach to indicator 
development on ICT in education, I 
want to give two examples from my 
own research where such a multilevel 
approach has become apparent. 


Two examples 

Example 1 — PILOT (project 
innovation in learning, 
organisation and technology) 

PILOT was the largest and most exten- 
sive project in Norway related to the 
pedagogical use of ICTin schools during 
the years 2000-04. The project was 
initiated by the Ministry of Education 
and Research, and a national agency 
(ITU) was responsible for coordinating 
the research work and research com- 
munities involved in the project. 

This project was part of upscaling of 
activities on a national level using new 
digital technologies, from a few inno- 
vative teachers and schools towards 
whole school communities and includ- 
ing many schools. Some 120 primary 
and secondary schools in nine regions 
of Norway took part in this four-year 
research and development project 
based on interventions concerning the 
educational use of ICT and developing 
a framework within whole school set- 
tings. The aim of the project was: ‘to get 
the participating schools to develop the 
pedagogical and organisational oppor- 
tunities afforded by the use of ICT, and 
to develop and spread new knowledge 
on this subject’. The research design 
was structured with a quantitative 
part (pre-post) and a qualitative part 
(during). 

In the initial phase, infrastructure 
and technological challenges were in 
focus. However, in the second part of 
the project, the focus was much more 
on various pedagogical approaches 
to education. This was due in part to 
the fact that the use of technology had 
become more common in everyday life 
at many of the schools, and in part to 
the fact that technology could not be 
used as a helpful aid until the proper 
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conditions had been established. In 
other words, the schools spent time 
restructuring the school day so that 
they could benefit from the educational 
opportunities that ICT represented. 
A number of the regions reported a 
positive impact on the pupils’ learning 
achievement with respect to academic 
performance, motivation for learning 
and changes of the subject content 
through the use of digital learning 
resources. 

Results from this project showed that 
schools handled the challenges of 
change and the introduction of ICT as 
a new object in very different ways. 
Four typologies of schools were identi- 
fied according to two dimensions, one 
going from unsystematic versus sys- 
tematic in the way school communities 
worked towards school development, 
and another going from being devel- 
opment oriented in the school culture 
towards being dominated by resist- 
ance towards change (Erstad, 2004). 

Findings on different levels 

Use of technology 

Writing activities of pupils and teach- 
ers increased. 

• There are differences between pupils 
and teachers in relation to how they 
use ICT. Many teachers do not rec- 
ognise the pedagogical opportuni- 
ties that the technology affords. 
Pupils and teachers experienced an 
increase in their competencies in 
using ICT. 

- Use of digital portfolios provided 
many pedagogical opportunities, for 
example in connection with parent/ 
teacher meetings. 

Pupils, teachers and head teachers 
were positive towards the use of ICT 
in teaching throughout the project. 


Technological problems dominated 
the project during the first year, but 
were then resolved for most schools. 

At the school level 

Those schools which worked holis- 
tically achieved the best results in 
terms of school development and 
ICT was also more integrated into 
pedagogical practices. 

• There were divisions and conflicts in 
the teaching staff at most schools, 
but there were major variations in 
relation to how this was handled by 
the school leaders. 

Over the course of the period, 
a majority of the PILOT schools 
attained a larger contact network 
vis-a-vis the local community. 

• The significance of easing the tran- 
sition between the school levels was 
documented. 

School administrators 

PILOT as a project involving the 
whole school community was chal- 
lenging for school administrators. 
The majority of principals reported 
that the school had initiated changes 
in activities in the school organisa- 
tion due to the integration of ICT, 
such as no longer using paper for 
sending out messages and instead 
putting them on the local network. 

Pupils and teachers 

PILOT focused on the importance 
of professionalising the teaching 
profession. 

• Pupils want a teacher who is a clear 
academic and pedagogical leader 
even though ICT is used more 
extensively. 

■ The majority of teachers were uncer- 
tain about the pedagogical use of 
ICT. 
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Teachers believed that ICT has a 
positive effect on pupils' perform- 
ance, that it creates more flexibil- 
ity and differentiation, and that this 
tendency was amplified during the 
course of PILOT. 

After the introduction of ICT, teach- 
ers experienced a positive change in 
their work day that intensified during 
the PILOT period. 

There is often a small group of 
enthusiastic teachers running the 
activities. Activists are important. 

Sustainability 

The school leaders reported that 
they would continue the restructur- 
ing efforts and ICT work after PILOT 
had finished. 

Learning communities help create 
a basis for and support change 
processes. 

• in the majority of schools, the PILOT 
activities gained a stronger local 
foundation. 

Example 2 — Networks of 
learning 

The Ministry of Education decided 
in 2004 to establish a national pro- 
gramme for school development and 
ICT called ‘networks of learning’, in all 
regions of Norway. It was structured 
with 10 schools in each network, from 
primary to upper secondary levels, 
and with one teacher training college 
leading each network. About 600 dif- 
ferent schools were involved in the 
project until June 2009 when the pro- 
gramme ended. 

Organising by networks is an alter- 
native to a hierarchical and rational 
goal-oriented approach, where the 
main aim is to develop the collective 
competence in the group of members. 
Strategies for collaboration, develop- 
ments of trust and support in addition 


to the advancement of knowledge and 
experiences are important. Of course, 
the challenges for optimal function of 
such networks are huge and it might 
be difficult to find the right balance 
between a strong leadership for devel- 
opment and stimulating initiatives 
among participants where leadership 
is more invisible. Networks are by 
definition decentralised, which makes 
leadership and division of responsibil- 
ity and labour a challenge. The main 
focus is on the role technology has in 
supporting and building networks for 
learning. 

An important aim of the programme 
has been diffusion of innovations to 
a large number of schools, through 
small funds and incentives. In the 
different reports during the last four 
years, teachers and school leaders 
report that the economic funds have 
not been the most important incentives 
for participating. Rather it is the possi- 
bility of working with others in building 
capacities that make both each school 
but also the collective efforts in each 
network stronger. 

Starting up 

The first year of the programme was 
dominated by a lot of insecurity, 
unclear definitions of responsibility on 
different levels (locally, regionally and 
nationally) and technologies that did 
not work optimally between schools. 
After the first year, the participating 
schools became more experienced, 
and the division of labour and 
responsibility was made clearer, which 
created a platform to define a new 
phase of more strategic development. 
The intention of the programme was 
to build up capacities for learning 
and networking that could be further 
developed after the programme 
ended, implying a model for expansive 
learning and knowledge building. By 
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using a strategy of reflection on action, 
networks have been able to learn from 
the challenges and tensions during 
the first phase, for example in the 
way networks have become more 
focused in their work, concentrating 
on certain aspects of technology use 
and educational perspectives, instead 
of trying to be too broad in their 
approach. 

Experiences by school leaders 

Arecent report (Eliassen, Josendal and 
Erstad, 2008), based on a survey done 
among school leaders in the partici- 
pating schools, shows that the overall 
impression is that there is a very posi- 
tive attitude among both school lead- 
ers and school communities towards 
working together in networks in this 
way to build capacity for change. The 
school leaders further reported the 
following. 

■ The experiences of working more 
closely with the teacher-training 
colleges was inspiring and created 
better conditions for school devel- 
opment because they had someone 
from outside their own community to 
follow them over time and give feed- 
back on activities both online and 
offline. 

• Participation in this programme 
increased the amount of discussions 
about educational issues, on school 
development and the use of ICT. 

• In general they have positive experi- 
ences of working with other schools, 
mainly in smaller networks (mini- 
networks) between teachers from 
different schools or with one or two 
other school communities. 

■ The use of ICT both for networking 
and in educational settings improved, 
but not as much as expected at the 
beginning of the programme. 


Diversity of network models 

A qualitative study was done towards 
the end of the programme, doing inter- 
views with different participants in 
several networks. This study shows a 
broad diversity of experiences across 
different networks (Skogerbo, Ottestad 
and Axelsen, 2007), both related to the 
way networks work with different issues, 
and to the different ways networks are 
organised. The development process of 
the networks became more focused and 
meaningful for the participants when 
each network defined a specific issue or 
theme to concentrate on. For example, 
some schools focused specifically on 
multimodal texts and how teachers and 
students could use specific technol- 
ogy within different subject domains. 
Others looked at how schools in a net- 
work could use a learning management 
system (LMS) to support collaboration. 
In this way, the networks also gained a 
clearer idea of the possible potential of 
using ICT for certain purposes, which 
increased the reported time spent with 
using ICT at these schools. 

An interesting outcome so far has 
been to see how networks organise 
themselves in different ways, often 
based on local interests and expe- 
riences. Some keep a hierarchical 
model where the teacher training col- 
lege in the network is taking the lead. 
Others are organised in a much more 
horizontal way, with different schools 
contributing in different ways and 
taking responsibility, without any spe- 
cific overall leaders. One success cri- 
teria for many networks has been the 
development of mini-networks within 
the larger network. In this way, teach- 
ers within science education could 
develop their own network based on 
their interests and needs, or principals 
could have their own network. These 
mini-networks have shown interesting 
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developments of knowledge building, 
focusing on how to build experiences 
and knowledge together over time. 

The working method chosen in most 
networks was a combination of meet- 
ings where participants met face to 
face, and online collaborative efforts. 
The physical meetings turned out to 
be very important for the networks, 
because they got time to discuss and 
reflecttogetherandto bring uptensions 
and problems in the developmental 
process at the schools, as part of the 
expansive learning processes. The 
teachers and school leaders reported 
that these meetings had an important 
function to make the networks evolve 
as communities of learning. 

Limitations and challenges 

A meta-evaluation of experiences and 
activities show that there are important 
challenges with this kind of development 
work involving many actors on different 
levels of the education system. This 
also indicates that this programme has 
limitations related to the initial objec- 
tives and ambitions of the programme 
from policy level. Some important chal- 
lenges have been the following. 

To get teacher training colleges 
to become more development ori- 
ented. Many of these colleges have 
huge challenges in keeping up with 
developments within schools, espe- 
cially on using ICT. 

■ Many teachers report lack of enough 
time to follow up development work 
as intended. The way schools are 
organised and the daily duties of 
teachers make development efforts 
come on top of everything else. 

• Almost all networks have reported 
difficulties in keeping up activities 
between face-to-face meetings. 
Online activities to stimulate devel- 
opment work are difficult without 


special planning. In some mini- 
networks, online collaboration has 
worked better because they have a 
more focused approach and a clearer 
understanding of why they use 
online resources for networking. 

Schools that already had experi- 
ence with using ICT reported that 
they felt that they gave more than 
they got in return. This is due to 
the way networks were organised, 
where schools with more experi- 
ence in using ICT should work with 
schools with less experience in this 
area, but which might have experi- 
ence in other areas that they could 
bring to the collaboration. 

Commitment of school leaders and 
school owners to make sure of sus- 
tainability over time. 

Dimensions of indicators 

So how might these examples and 
the discussion above help in devel- 
oping a multilevel approach on indi- 
cators about the impact of ICT on 
education? Most importantly, what is 
described above shows the necessity 
of understanding ICT and its impact 
on education on different levels. The 
synergy of different levels is the basis 
for change and development in both 
projects, where ICT is both a catalyst 
for change and a new cultural tool for 
enhancing student learning. 

This implies a higher degree of com- 
plexity in developing indicators. 
However, the results from studies like 
the ones mentioned above show that 
schools that define ICT as important 
on different levels of the organisation 
and have a strategy for how the whole 
school should orient itself towards 
the use of ICT are more successful 
in using ICT for educational purposes 
than other schools. 
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Below, I present some key compo- 
nents that are important as sets of 
indicators to measure the impact of 
ICT on education. Again, I will mainly 
build on projects and developments in 
Norway. Perspectives on digital litera- 
cies/competencies are seen here as 
something that frames these sets of 
indicators, something aggregated that 
relates to all indicators in one way or 
another. 

Across levels: digital 
literacy as the framing 

Digital literacy relates to both an ability 
to operate technological applications 
and to use technology to accomplish 
personal and collective needs. In this 
sense, it raises important questions 
about new digital divides in the popula- 
tion, between the ones who know how 
to operate the technology and the ones 
who do not, and between the ones who 
use the technology to gain relevant 
knowledge for education and the ones 
who use it for other purposes. 

This implies that we constantly have 
to ask the more general question of 
what it means to ‘read’ and 'write' in 
a culture, and thereby how we learn 
(Pahl and Rowsell, 2005). In their 
Handbook of literacy and technology: 
transformations in a post-typographic 
world, David Reinking et al. (1998) 
present several perspectives on how 
the development of digital technolo- 
gies changes conceptions of text, of 
readers and writers and ultimately of 
literacy itself. This implies that digital 
literacy relates to changes in traditional 
cultural techniques such as reading 
and writing, and yet meanwhile opens 
up new dimensions to what it means 
to be a competent reader and writer 
in our culture, and the institutions that 
support these processes. 


In her book Literacy for sustainable 
development in the age of information 
(1999); Naz Rassool presents an over- 
view of different debates on literacy 
in recent decades. Her point is that 
research perspectives on technology 
and literacy need to reconceptualise 
power structures within the informa- 
tion society, with an emphasis on ‘com- 
municative competence' in relation to 
democratic citizenship. Digital technol- 
ogies create new possibilities for how 
people relate to each other, how knowl- 
edge is defined in negotiation between 
actors and how it changes our concep- 
tion of learning environments in which 
actors make meaning. Empowerment 
is related to the active use of differ- 
ent tools, which must be based upon 
the prerequisite that actors have the 
competence and critical perspective on 
how to use them for learning. Literacy, 
seen in this way, implies processes of 
inclusion and exclusion. Some have 
the skills and know-how to use them for 
personal development, but others do 
not. Schooling is meant to counteract 
such cultural processes of exclusion. 

One report on conceptualising digital/ 
ICT literacy often referred to is Digital 
transformations: a framework for ICT 
Literacy (ETS, 2002) written by a team 
of experts for the Educational Testing 
Service in the USA. In this report, they 
identified some key concepts of what 
they called ICT literacy. One interpre- 
tation of such key concepts can be the 
following (my elaboration based on 
ETS). (See Figure 1). 

This consists of more general com- 
petencies (communicate, create, 
access, information handling, critical/ 
analytical) that are not connected to 
specific subjects in school or specific 
technologies. They can be taught and 
are not only related to what is learned 
in school settings, but also to situa- 
tions outside the school. 
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Basic skills 

Be able to open software, sort out and save information on the com- 
puter and other simple skills in using the computer and software. 

Download 

Be able to download different information types from the Internet. 

Search 

Know about and be able to get access to information. 

Navigate 

Be able to orient oneself in digital networks, learning strategies in 
using the Internet. 

Classify 

Be able to organise information according to a certain classification 
scheme or genre. 

Integrate 

Be able to compare and put together different types of information 
related to multimodal texts. 

Evaluate 

Be able to check and evaluate if one has got the information one 
seeks to get from searching the Internet. Be able to judge the qua- 
lity, relevance, objectivity and usefulness of the information one has 
found. Critical evaluation of sources. 

Communicate 

Be able to communicate information and express oneself through 
different mediation means. 

Cooperate 

Be able to take part in net-based interactions of learning, and take 
advantage of digital technology to cooperate and take part in 
networks. 

Create 

Be able to produce and create different forms of information as 
multimodal texts, make web pages and so forth. Be able to develop 
something new by using specific tools and software. Remixing diffe- 
rent existing texts into something new. 


Figure 1 . Key concepts of ICT literacy (my elaboration based on key concepts in the ETS report) 


Other frameworks have used ‘dig- 
ital competence’ as an overall term. 
One example is the working group 
on key competences of the European 
Commission, ‘Education and training 
2010’. This programme identifies dig- 
ital competence as one of the eight 
domains of key competencies, defin- 
ing it as ‘the confident and critical use 
of information society technologies 
for work, leisure and communication. 
These competencies are related to log- 
ical and critical thinking, to high-level 
information management skills and to 
well-developed communication skills. 
At the most basic level, ICT skills com- 


prise the use of multimedia technology 
to retrieve, assess, store, produce, 
present and exchange information, 
and to communicate and participate in 
networks via the Internet.’ (European 
Commission, 2004, p. 14). Digital 
competence in this framework encom- 
passes knowledge, skills and attitudes 
related to such technologies. 

As shown in this section, there are dif- 
ferent frameworks to relate to in our 
understanding of digital literacy/com- 
petence which relate to different levels 
and issues. However, the key challenge 
is to go deeper into the implications of 
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increased use of new technologies in 
educational practices. 

Different levels combined 

Most often, impact has been related to 
the individual level. The interest, espe- 
cially among policymakers, has been 
in student outcomes when using ICT. 
However, this is not as easy to detect 
as it might seem. It has been prob- 
lematic to define clear effects and out- 
comes, first of all because it is difficult 
to isolate the effects of ICT itself since 
most schools change many aspects of 
their teaching and learning practices 
when they start to use ICT. 


A more fruitful approach would be to 
study impact on different levels and look 
at co-variation between levels. This will 
give a broader and richer understand- 
ing of impact that is also closer to the 
experiences of schools. 

One way of defining indicators on dif- 
ferent levels is to describe them on 
macro, meso and micro levels. Two 
of the levels of indicators mentioned 
in Figure 2 are on the macro level 
(national, local). The meso levels 
would be the institutional and learning 
environments. The micro levels focus 
on teacher and student practices and 
outcomes (collective and individual). 
Below is an attempt to bring together 
different levels and different contexts 



Figure 2: Different levels of understanding the impact of ICT in education 


Multilevel approach to address complexity 


where ICT plays a role for education 
and learning. 

Indicators and levels 

For each level, a set of indicators is of 
relevance, and for some levels indi- 
cators of impact are well established, 
while for others the development of 
indicators has been limited. 

Different levels and indicators also 
imply different methods of collecting 
information on the possible impact of 
ICT on education. Monitoring of impact 
can be done in several ways as a com- 
bination of quantitative and qualitative 
methods. 

National level 

Impact on a national level deals with 
key factors of importance for how ICT 
is implemented in the school system in 
different countries. This is most of all 
related to the ways countries define 
ICT as of importance in educational 
development. This is to go beyond the 
policy slogans about the importance of 
ICT in itself and a technological deter- 
minism, and focus more on the con- 
crete steps taken by policymakers in 
different countries. The methods used 
for such indications of impact could 
be analysis of policy documents and 
monitoring through national surveys 
of developments within the education 
system. Some key indicators on this 
level are as follows. 

Curriculum development: In many 
countries, ICT is mentioned in curric- 
ulum documents, but it differs in what 
way and to what extent. In most coun- 
tries, curricula are important in the 
way they frame the education system 
and the practices taking place within 
these systems. For example, in my 
own country (Norway), digital literacy 
has been written into the national 


curriculum as of 2006. From a former 
situation where ICT was mentioned 
as a tool that might be integrated into 
the classroom, the new curriculum 
states that ICT has to be used in all 
subjects and on all levels of compul- 
sory schooling. There has thus been 
a marked impact on the curriculum. 
Infrastructure/access: In most coun- 
tries during the last decade there has 
been a prime focus on making com- 
puters and Internet connections avail- 
able to educational institutions. This 
has partly been a national responsi- 
bility by ministries and other national 
agencies, and is expressed in different 
national documents and action plans. 
Some countries have also adopted 
instruments to monitor progress in 
this area, which specify the ratio of 
computers and Internet access per 
students and teachers. A critique has 
surfaced in recent years about the 
focus on implementation of technol- 
ogy in the education system for too 
much technological determinism. 
Standardisation: Many countries 

have started work on standardisation 
of technological solutions. The ISO 
standard has been implemented in 
several European countries for the 
coordination of technological devel- 
opments and to make use more 
accessible across different technolo- 
gies and platforms. This has become 
an important part of technological 
strategies on national levels, as an 
indication of developments within 
ICT and education systems. 

Digital learning resources: National 
initiatives to stimulate the production 
of digital learning resources have 
been important, yet problematic, in 
many countries. As such, they are 
an important indicator of progress 
on a national level, because they 
are important for how teachers 
and students use ICT in education. 
Publishing companies have invested 
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in technological developments to 
develop different learning resources 
beyond the book. Yet investments 
have not always made a profit and 
such companies are often reluctant 
to make the necessary investments. 
This has also raised issues about 
public and private collaborations to 
develop such resources on a sys- 
tems level for education. 

• Use: Some countries have instru- 
ments to follow the actual use of ICT 
on different levels within the educa- 
tion system. This is to get a national 
overview of implications of invest- 
ments and implies a set of indicators 
to be developed at a national level 
to map how ICT is used on different 
levels and subjects in order to com- 
pare and see developments. 

Local level 

Strategies: Important on a local level 
is the extent to which local authori- 
ties develop strategies, expressed 
in different kinds of documents, to 
give a direction for the implementa- 
tion and use of ICT in education. It 
varies a lot as to how well such docu- 
ments and local policies are devel- 
oped and used. Some are too vague 
and contain unrealistic intentions and 
visions; others have clear objectives 
and implementation plans. 

• Infrastructure/access: Even though 
there are national policies concern- 
ing the implementation of infrastruc- 
ture, it varies to what extent this 
is followed up on a local level. It is 
therefore necessary to develop indi- 
cators that track the implementation 
of infrastructure on a local level. 
Support: Another important aspect 
concerning impact on a local level 
is support structures, both for imple- 
mentation of technology and guide- 
lines for use. Local authorities have 
been important in many countries 
in developing such support struc- 


tures, which are important especially 
to secure the use of ICT among 
teachers. 

Institutional level 

■ Leadership: On the institutional level, 
the leadership at the school is impor- 
tant in creating the setting for ICT 
use. This of course relates to the 
implementation strategies developed 
by national and local authorities, but 
also to how the leadership gives 
direction to certain developments. 
This also concerns how the school 
and the leadership at the school 
make the strategies for school devel- 
opment with the use of ICT explicit. It 
often varies how the school leader- 
ship manages to develop strategies 
that have real implications on a prac- 
tical level. Another indicator concern- 
ing leadership could be how schools 
use ICT as an administrative tool. 
School culture: Each school is differ- 
ent from another due to differences 
in leadership, the teacher community, 
the local community of the school, 
the student population and so forth. 
School culture relates to the daily 
life of each school. The school cul- 
ture influences the way ICT is imple- 
mented and used in the school. As 
shown in the PILOT project above, 
some schools see ICT as a catalyst 
for change while others are much 
more sceptical towards ICT. 
Collaboration: This could be an indi- 
cation of the ways teachers collabo- 
rate and share experiences in order 
to build up competencies in using 
ICT. Collaboration could also be 
between schools, between school 
leaders in a community, or between 
students nationally and internation- 
ally. The point is that this is often an 
indication of how schools use ICT as 
a tool for collaboration. 
Reorganisation:An indication ofimpact 
on the institutional level also relates to 
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the extent to which schools start to 
reorganise their practices due to the 
implementation of new technologies. 
For example, that the introduction of 
laptops makes it difficult to uphold a 
traditional classroom setting. 

Teacher education level 

Teachers’ ICT competence: To what 
extent teacher education has imple- 
mented courses and strategies 
towards the increased competence 
of teachers in using ICT is an impor- 
tant part of educational development 
and change. This could be seen as 
ICT literacy indicators for teacher 
education, and of how teachers are 
prepared to face the challenges in 
their practice as teachers. 

Teaching methods: This point relates 
to the training of teachers in differ- 
ent methods of using ICT and digital 
resources. This implies a change 
within teacher training colleges in the 
way the teaching profession might be 
performed using ICT. 

• Written strategies: For schools, 

teacher training colleges also need 
written strategy documents that give 
direction and indications of change. 

Learning environment level 

■ ICT use: The ways ICT is actually 
used within learning environments. 

• Flexibility: At school level, the tradi- 
tional classroom might be changed 
into a more flexible understanding 
of learning spaces and rooms, big or 
small, which are used for learning. 
The technology might push for this. 

• Online/offline: Learning environ- 

ments might also be thought of as 
a combination of face-to-face offline 
interaction, and online environments 
for learning activities. This also indi- 
cates an opening up of the learning 
environment to the outside world. 

• DLRs used: This concerns the 
extent to which digital learning 


resources are used within the learn- 
ing environment. 

Assessment: To what extent assess- 
ment procedures are changed. Flow 
teachers and students use sum- 
mative and/or formative ways of 
assessment. 

Collective level 

Collaborative work: This point is an 
indication of how the use of ICT might 
stimulate more collaborative work 
among students, and that project 
work becomes more prevalent in 
schools. 

■ Sharing content: To what extent stu- 
dents and teachers upload content 
produced in schools to the Web and 
sharing it with others. Or the extent 
to which they reuse content that they 
find on the Web as part of their own 
learning activities. 

Individual level 

Outcomes: Different indications of 
the outcomes of ICT use on the indi- 
vidual level, both in a summative and 
a formative way related to learning. 

• Knowledge building, problemsolving: 
The ways in which ICT stimulates 
knowledge building and problem 
solving among students, assessed 
by performance assessment. 

■ ICT competencies: The differences in 
ICT competencies among students, 
the digital divide. 

These are just some examples of indi- 
cators that might be thought of on dif- 
ferent levels. Some indicators overlap 
on different levels; others are unique 
for specific levels. When we have this 
more holistic view of indicators on dif- 
ferent levels, we might see better how 
they are important in different ways on 
different levels. Some of these levels 
and indicators are directed towards 
preconditions for use of ICT, some 
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towards the framing of such use and 
some towards the actual use and 
outcomes of such use. Indicators on 
national and local levels are primarily 
preconditions for use in the way they 
create the platform and the basics for 
use by providing the technology. The 
framing relates to the institutional level, 
teacher education and the learning 
environment, which create conditions 
for how ICT will be used in educational 
settings, while the collective and indi- 
vidual aspects relate more directly to 
the use of ICT itself and to outcomes 
of such use. 

Implications 

In specifying indicators of ICT in edu- 
cation, the argument in this article has 
been to draw different levels together 
in order to get a fuller and wider under- 
standing of the role of ICT in our edu- 
cation system. As stated, this is not an 
easy task, but the risk of reducing the 
complexity of impact of ICT on our edu- 
cation system is that we only see a part 
of the picture, and that we do not see 
how things are interconnected. 

Such a multilevel approach has implica- 
tions for policy, practice and research. 

• Policy: Policymakers need to take into 
consideration how the system levels 
interconnect with the practice levels 
in their understanding of impact. My 
impression is that policies within this 
area have moved beyond simple 
technological determinism, believ- 
ing that technology itself will create 
change, towards an awareness of 
the complexity involved in drawing 
up policies for ICT in education. Still, 
the understanding of impact is often 
drawn towards simple outcomes 
on the individual level. A multilevel 
approach might give a more realistic 
understanding of how impact is inter- 
related on different levels, thereby 


avoiding reducing ICT in education to 
a question of whether students learn 
better now than before. Change and 
outcome is about the system of edu- 
cation and how students learn is con- 
nected to teachers’ competencies 
in this area, about the assessment 
system, about the available digital 
learning resources and so forth. 
Policymakers can develop strategies 
for systems of indicators and col- 
lection of such data that will provide 
them with the necessary tools for 
creating capacity for further develop- 
ment within this area. 

■ Practice: In order to stimulate use 
of ICT in educational practice, we 
need a better understanding of the 
interrelationship between different 
levels, and how each of them might 
strengthen or hinder changes within 
educational practices. It is the impact 
on the practical level that is of impor- 
tance, but that level is dependent 
on developments on other levels, 
like school leadership, digital learn- 
ing resources, curriculum develop- 
ment and so forth. Teachers and 
students need a framework that 
stimulates change and development. 
Perspectives on digital/ICT literacies, 
for example, have real implications 
on a practical level in the way this 
term applies to certain learning objec- 
tives using ICT. In addition, it relates 
directly to several other levels. 
Research: There is a need for more 
research that manages to grasp the 
complexity of the matters mentioned 
above. One example given in this 
article is activity theory developed 
by Yrjo Engestrom, but we need 
more development in this area to be 
able to develop analytic concepts 
and research tools that can help us 
research such a multilevel approach 
to the impact of ICT on educa- 
tion better than we are able to at 
present. 
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Abstract 

In this article , a description of educational monitoring will be provided. This consti- 
tuted the background for a study about monitoring ICT in primary and secondary 
education in the EU (') (see Chapter IV: Indicators on ICT in primary and second- 
ary education). First the function of monitoring for policymaking will be described, 
showing that educational monitors in general can have different functions, and 
the concepts of policy goals, indicators, instruments and data will be introduced. 
A distinction can be made between international, national and school monitor- 
ing. This is followed by a description of the main steps involved in designing and 
conducting international comparative educational monitors, sketching a number 
of dilemmas for which solutions need to be sought. This is followed by a review of 
methodological issues in international comparative monitoring. 


Functions of monitoring 

Monitoring can be defined very broadly 
as ‘the act of periodically/continu- 
ously observing something’. The act 
of observation will be called ‘assess- 
ment’ further on and hence regular 
assessment equals monitoring. An 
educational monitor is thus ‘assess- 
ment of education and how it is devel- 
oping overtime'. This definition is fairly 
neutral and could, in certain situa- 
tions, when explicit targets are set, be 
translated into ‘assessment of educa- 
tion in order to determine if standards 
are met’. Educational monitoring can 
be focused on many different charac- 
teristics of education, such as input, 
■processes and learning outcomes and 


(') This x study was financed (at a cost of 
EUR 122 200) by the European Commission. 
Contract EACEA-2007-3278. Opinions presented 
in this chapter do not reflect or engage the 
Community. © European Commission 


many different methods can be used 
for collecting observations. Qualitative 
and quantitative methods can be dis- 
tinguished. In this study, the main 
focus is on quantitative methods that 
allow for comparisons between coun- 
tries and, hence, imply statistical gen- 
eralisations to the educational system 
at large. 

A distinction can be made between 
national and international monitors. 
National educational monitors are 
meant to draw conclusions about 
changes that take place in educational 
systems over time, which implies that 
the observations are collected in 
such a way that they are comparable 
over time. International comparative 
educational monitors offer possibilities 
to interpret the state of the art and / 
or changes over time in one country 
with reference to changes in other 
countries, provided that the measures 
that are used are internationally 
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comparable between countries and 
over time. 

A fairly recent development is school 
monitoring, whereby schools keep 
track of their developments (some- 
times in comparison with other 
schools) for evidence-based school 
policymaking. A full multilevel monitor 
would be a system in which interna- 
tional, national and school monitoring 
are integrated. 

Monitoring in general can be con- 
ceived as regular assessments that 
are part of a cyclic policy process 
that consists of a number of steps, as 
shown in Figure 1. Monitoring implies 
a regular repeat of step 2 (Figure 2). 

Figure 2 concerns a very general 
model that can be applied in many 


different settings, for instance at the 
international (worldwide, regional), 
national, school and even individual 
level. 

Given the purpose of our study, we 
will further focus mainly on the inter- 
national level and will describe below 
in more detail each of the steps that 
are distinguished in Figure 1, in par- 
ticular in terms of what is required in 
each of these steps, which concepts 
are relevant and which questions and 
dilemmas will be faced. 

1. Policy goals 

Whereas national monitors are focused 
on policy goals that are relevant for one 
country's stakeholders, the group of 
stakeholders is larger for international 
monitors and the participating countries 



Figure 1: Steps in evidence-based policy cycle 
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need to decide first on which common 
goals a monitor should be focused. An 
example of a common goal might be 
To connect all schools to the Internet’. 
A dilemma in establishing common 
goals is that some goals may be highly 
relevant in some countries (e.g. those 
which are just starting to connect to the 
Internet), but not or not yet relevant in 
other countries (e.g. those which have 
already realised this goal). We will call 
this ‘goal disparities'. 

What can also happen is that certain 
common goals have a short ‘lifetime’, 
so that they were perhaps highly rel- 
evant in a certain time period, but 
were no longer so later on (for ex- 
ample because the goals have been 
reached). In relation to ICT particularly, 
where rapid technological develop- 
ments are taking place, this is an issue 
of special concern (in this respect the 
notion of ‘life expectancy’ of indicators 
becomes relevant). 

Once common goals have been 
established, indicators for monitor- 
ing the progress towards these goals 
need to be defined. If goal statements 
are very concrete, as in the example 
above, this may be relatively easy to 
do, such as ‘the percentage of schools 


that have a connection to the Internet’. 
However, when the goal statements 
are fairly global, as is often the case in 
international consensus-building proc- 
esses (e.g. ‘provide all students with 
access to the Internet’), a number of 
different indicator definitions may be 
needed (e.g. number of Internet con- 
nected computers per 100 students, 
connection speed, etc.). 

A serious problem in defining indicators 
concerns their ‘comprehensiveness’, 
which is the extent to which they ade- 
quately cover the domain that is implied 
by the goal statements. Monitors can 
potentially have quite serious (unin- 
tended) conservative impacts on edu- 
cational policymaking if the compre- 
hensiveness is low. This can occur if, 
for instance, they do not cover relatively 
new competencies, but rather focus on 
traditional competencies of students. 
For example, suppose that the use of 
ICT leads to a slight decrease in math- 
ematics skills (for which an indicator is 
available), because as a result of stu- 
dents’ autonomous working less con- 
tent can be covered. If, at the same 
time, a high increase in communication 
and studying skills (for which no indi- 
cators are defined) occurs, this posi- 
tive effect would remain unnoticed and 
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there would be a chance that ICT use 
in mathematics would be discouraged. 
In this respect, the notion of ‘holistic 
monitoring’ is relevant. International 
comparative assessments may have 
a big impact on education. Recently a 
consortium of Cisco, Intel and Microsoft 
concluded that, in order to reform edu- 
cation, the current prevailing inter- 
national comparative assessments 
would have to be changed. 

For practical reasons, the number of 
indicators that can be addressed in an 
assessment is limited (see point 2). 
Therefore establishing priority needs 
is an essential aspect of step 1 . 

An important distinction in Figure 1 is 
between ‘primary’ and ‘secondary’ indi- 
cators (sometimes also called respec- 
tively key indicators and background 
or explanatory indicators). Primary 
indicators are those that are featured 
as the main focus of an assessment; 
for instance when it concerns PISA or 
IEA-TIMSS-PIRLS, primary indicators 
concern the test results in mathemat- 
ics, science and/or reading, which 
are usually the first to be featured 
when statistical reports from these 
international monitors are released. 
Secondary indicators are used to 
throw further light on the test results, 
for instance by examining difference 
in outcomes between sub-populations 
in countries (e.g. boys and girls) or for 
analysing how the differences between 
countries can be explained. 

2. Assessment 

An international comparative assess- 
ment consists of collecting data in rep- 
resentative national samples on the 
basis of instruments (usually question- 
naires and tests) that contain opera- 
tionalisations of the intended indica- 
tors (from step 1). There are several 


issues and constraints that need to be 
considered when designing an inter- 
national comparative assessment. 
Firstly, as the instruments are adminis- 
tered to educational actors in schools 
(school leaders, teachers, students 
and sometimes parents) a serious 
constraint is the amount of time that 
can be asked from each respondent 
to answer the tests/questionnaires. 
Increasing the amount of time will lead 
to lower response rates, which then in 
turn would affect the quality of national 
statistical estimates that are based on 
the collected data. As the number of 
questions that can be included in ques- 
tionnaires is limited, this in turn has 
implications forthe number of intended 
indicators that can be included. Initial 
priority decisions can be made on the 
basis of a priori response time esti- 
mates. Further, during the process of 
operationalisation and piloting (when 
response-time estimates can be col- 
lected) it may appear that the number 
of intended indicators needs to be fur- 
ther reduced. 

An important issue concerning the 
operationalisation of intended indi- 
cators concerns costs. Developing 
completely new indicators is a time- 
consuming process, because empiri- 
cal evidence needs to be collected 
regarding the comparability, statistical 
quality and interpretability of the new 
measures. 

After the data are collected, ‘indicator 
statistics’ can be calculated. Forexam- 
ple, when an indicator definition might 
be ‘use of ICT', one of the indicator 
statistics might be ‘percentage of stu- 
dents using ICT daily at school’. If the 
same intended indicator was included 
in earlier assessments, another indi- 
cator statistic might be ‘increase of 
daily use of ICT at school between 
2000 and 2009’. 
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3. Evaluation 
and reflection 

Once the indicator statistics are 
available, the interpretation of the 
outcomes can start. This would be 
quite a straightforward process in 
cases where the common goals and 
intended indicators were phrased in 
operational terms (e.g. 'we expect that 
at least 90 % of students are using 
ICT daily in school’), which for obvi- 
ous reasons is hardly ever the case. 
Hence, this is usually a very tedi- 
ous process, in which many different 
groups of stakeholders have their say. 
It is not uncommon to observe that 
national researchers present well- 
qualified interpretations covering both 
strengths and weaknesses, while after 
the publication of the international 
reports, the media and certain groups 
of stakeholders interpret the outcomes 
like Olympic league tables, whereby a 
place lower than the top three is quali- 
fied as bad and reason for serious 
policy concern and action. This is the 
phase where the highest risk exists 
of eliciting conservative effects (see 
above), but many other potential falla- 
cies exist that could lead to unjustified 
interpretations of the statistics. 

Evaluation and reflection may lead to 
generating concerns and questions 
about potential causes of what has 
been observed. In general, it appears 
that in countries ranking high on the 
primary indicators, not many initiatives 
for follow-up activities will occur, while 
in countries that rank low, questions 
will be raised about potential causes 
of this outcome. 

4. Diagnosis 

Once the primary indicator statistics 
have raised concerns about the exist- 
ence of weaknesses in the education 


system, the need may arise to try to 
find out what the potential reasons are 
that could lead to interventions aimed 
at realising improvements. Existing 
international comparative monitors 
usually include quite a number of sec- 
ondary indicators that are intended 
to be used for explaining the differ- 
ences between countries and between 
schools and students within coun- 
tries. A common experience among 
researchers involved in the process of 
finding causes is that the set of second- 
ary indicators is too limited to answer 
concrete why questions that are posed 
after the data have been collected, and 
hence this often does not result in con- 
crete suggestions for policy interven- 
tions that could lead to improvement. A 
more fundamental problem is that the 
collected data do not allow for cause- 
effect analyses. At best they can result 
in strengthening or weakening par- 
ticular beliefs about cause and effects. 
Therefore some countries occasion- 
ally conduct additional research in 
order to find out whether handles can 
be found for improvement. In the past, 
one country (the Netherlands), scoring 
low on international reading tests, con- 
ducted in-depth analyses on the read- 
ing methods used in schools and con- 
cluded that these were no longer up 
to date. A change of reading methods 
took place and later it appeared that 
the international ranking had consid- 
erably improved, which strengthened 
the belief that the reading methods 
were among the potential causes of 
low performance. This is an example of 
qualitative follow-up of the international 
assessment. 

5. Interventions 

Throughout the world, many examples 
are available of policy actions that 
were undertaken as a result of the 
outcomes of international comparative 
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assessments. It seems safe to 
infer, on the basis of the continuing 
increase of participating countries in 
international comparative educational 
monitors (from 20 in the IEA studies 
in the 1980s to over 60 in the current 
IEA studies), that policymakers are 
becoming more aware of the potential 
benefits of international comparative 
educational monitors for evidence- 
based policymaking. 

It should be noted that interventions do 
not necessarily need to be top-down: 
if schools in a country could see how 
they perform on the primary indicators 
(by means of school monitoring) and 
make inferences about the existence 
of potential weaknesses and their 
likely causes, these initiatives might 
be designed and generated at school 
level. This approach is advocated in 
some EU countries. 

The policy cycle that is sketched above 
may help to illustrate several functions 
that international comparative moni- 
tors may have, such as: description 
(mirror), accountability, benchmark- 
ing, enlightenment, understanding 
and cross-national research. Some 
of these functions (such as bench- 
marking, monitoring, understanding 
and cross-national research) can be 
explicitly addressed by the research 
design, while other functions are more 
or less collateral (mirror, enlighten- 
ment). Monitors can help in the proc- 
ess of evidence-based policymaking 
by which decisions are based on facts 
rather than rhetoric. In this sense, 
monitors are also conceived as navi- 
gation tools. However, one should 
also be aware of potential resistance 
to participate in international compar- 
ative monitors, as these may be per- 
ceived as leading to undesirable influ- 
ences on educational policymaking. 


This may, in particular, be the case 
when it concerns ICT indicators. 

The main steps underlying the design 
and execution of a monitor can be 
summarised as follows. 

1. Establishing common objectives 

2. Defining indicators 

3. Operationalising indicators 
(= instruments) 

4. Drawing samples of respondents 

5. Collecting data 

6. Presenting descriptive results 

7. Generating questions for 
diagnosis 

8. Analysing data 

9. Making recommendations for 
interventions 

10. Making recommendations for 
revised/new indicators. 

Participation of EU 
countries in full-scale 
international comparative 
educational monitors 

Despite the needs for monitoring, 
resulting from benchmarks that were 
established after the open method of 
coordination was introduced follow- 
ing the Lisbon 2000 summit, the EU 
has no system in place for a full-scale 
regular monitoring of quantitative 
indicators of student skills in primary 
and secondary education. It there- 
fore needs to rely on data collected 
through other international organisa- 
tions, mainly the IEA (International 
Association for the Evaluation of 
Educational Achievement) and the 
OECD (Organisation for Economic 
Cooperation and Development). Both 
organisations conduct regular inter- 
national comparative assessments 
measuring (among other things) stu- 
dents’ skills in mathematics, science 
and reading. 
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The IEA has existed for over 50 years. 
As a non-governmental organisa- 
tion, it conducts large-scale quantita- 
tive assessment in mathematics, sci- 
ence, reading, civic education and ICT, 
amongst other things. The core studies 
(in mathematics, science and reading) 
take place roughly every four years 
and, since 2000, the assessments 
have also been conducted roughly 
each four years. In 2011, a combined 
assessment of mathematics, science 
and reading will take place. The OECD 
PISA assessment was conducted for 
the first time in 2000 and is run every 
three years. The core performance 
domains are mathematics, science and 
reading. The latest assessment took 
place in 2009 and is expected to be 
reported by the end of 2010. The next 
assessment is scheduled for 201 2. 

Since 2000, the majority of EU coun- 
tries have participated in the OECD 
assessments (PISA) and/or IEA 
(TIMSS and PIRLS, respectively 
mathematics/science and reading) at 
the primary and/or secondary educa- 
tion level. 

Core areas for monitoring 

For monitoring educational progress, 
at least three main core areas need to 
be considered, namely: 

• intended learning outcomes; 

• opportunities to learn (OTL); 

• competencies/attitudes of students. 

Definitions of intended outcomes 
are needed for steering educational 
processes that result in OTL, which 
in turn are supposed to influence the 
competencies and attitudes of stu- 
dents. Moreover, these definitions are 
needed to be able to construct tests 
for measuring the extent to which the 
intentions are realised. 


Intentions may be formally legislated 
in syllabi, examination standards or 
in the words of the IEA ‘intended cur- 
ricula’. These constitute the basis for 
guiding many educational processes, 
such as the content of the textbooks, 
teaching and learning activities in 
schools, the content of (in-service or 
pre-service) teacher training, etc. An 
analysis of these intentions is usually 
the basis for designing international 
comparative assessments that are 
currently run by international organi- 
sations, such as OECD (PISA) and 
IEA (TIMSS, PIRLS). These analyses 
may be based on extensive curricu- 
lum analyses (IEA) or expert opinions 
about what the important life skills 
are that students need to acquire in 
schools (OECD). The outcomes of 
such analyses constitute the basis for 
developing the content specifications 
for the instruments that are used to 
measure educational outcomes (e.g. 
in the cognitive domain, such as math- 
ematics, science and reading, but also 
affective, e.g. learning motivation), 
whereas on the other hand these 
content specifications can also be 
used for measuring the opportunities 
that schools offer to students to learn 
these contents. Educational moni- 
toring that would only be focused on 
these three core concepts would allow 
educational actors to make a limited 
number of inferences, such as: 

• for national monitors: 

— whether intentions, OTL and out- 
comes are changing over time, 

— whether discrepancies exist 
between intentions and OTL, 

— whether inequities exist between 
sub-populations of students and 
how these are changing over 
time; 

• for international monitors: 

— the same as for national moni- 
torsbut with enhanced possi- 


47 


Chapter II — State of the art 


bilities to interpret the national 
observations with reference 
to what is happening in other 
countries. 

Although such inferences are impor- 
tant as a first step towards understand- 
ing educational progress, they would 
offer insufficient handles for undertak- 
ing policy actions for remediation. 

Therefore it is necessary in educa- 
tional monitors to also address con- 
cepts that are (politically) malleable 
and which relate to areas that are 
believed to influence OTL and out- 
comes. For these concepts (earlier we 
referred to secondary indicators), an 
almost endless variety of candidates 
could be generated, such as: 

• competencies of teachers; 

• number of hours in the timetable 
scheduled for certain OTL areas; 

• availability and quality of learning 
materials; 

• instructional methods applied; 

• school organisation and quality of 
leadership; 

• class climate; 

• examination standards. 

For the study that formed the basis of 
this chapter, the question was how ICT 
fits into the picture sketched above. 
ICT can be conceived as a transversal 
issue as well as a subject area. 

When ICT is a subject area (such as 
mathematics and science in existing 
international comparative monitors) 
the previous concepts could be trans- 
lated into, for example: 

• core concepts, such as: 

— the intentions (formally legislated 
or informally adhered to) with 
regard to ICT literacy, 

— the OTLs for learning about and 
learning with ICT, 


— the ICT-related competencies of 
students; 

• instrumental concepts, such as: 

— the competencies of teachers 
about ICT (technical ICT literacy) 
and the use of ICT (pedagogical 
ICT literacy), 

— the number of hours scheduled 
for learning about ICT, 

— the availability and quality of ICT 
learning materials. 

When ICTis conceived as a transversal 
issue, the concepts mentioned above 
could be considered all instrumental. 

Methodological issues in 
international comparative 
monitoring 

International comparative educational 
monitors (later in this section abbrevi- 
ated to ICEMs) are designed in such 
a way that high-quality data are col- 
lected that allow for generalisations to 
the defined national target populations 
and for comparisons between coun- 
tries. Several aspects that need to 
be considered in the design of these 
monitors are described in the sections 
below. 

Measurement 

In order to be able to make statements 
about the concepts (and derived indi- 
cators) underlying the assessment, 
measures are needed that can be used 
for statistical generalisations. With 
regard to measurement, a main dis- 
tinction that can be made is between 
‘what’ and ‘whom’ is measured. 

‘What’ refers to the constructs that are 
materialised in instruments. Typically 
in ICEMs, which are targeting stu- 
dents, the following types of instru- 
ments are distinguished. 
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• Context instruments: for collecting 
information about school external 
conditions (e.g. funding, regula- 
tions, curriculum). For instance, to 
what extent do curricula prescribe 
the use of ICT? 

• School instruments: containing 

questions about school character- 
istics (e.g. organisation, manage- 
ment, school policies). For instance, 
how many computers are available 
in schools or what is the vision of 
school leaders about desirable ped- 
agogical approaches using ICT? 

• Teacher instruments: containing 

questions about instructional prac- 
tices. For instance, to what extent 
do teachers use ICT for testing 
students? 

• Student instruments: tests for meas- 
uring achievement and student 
questionnaires about activities and 
background. For instance, how often 
did you use a computer for learning 
mathematics? 

Development of instruments 
The construction of international 
instruments is usually a very time-con- 
suming activity in which many steps 
are specified that are all intended to 
improve the quality of the assess- 
ments. Some of these activities are 
described briefly below. 

• Involvement of international experts. 
At the start of international assess- 
ments, committees of experts with a 
good reputation in the areas that are 
tested are formed. These commit- 
tees review items in order to guaran- 
tee that they represent the content 
area. 

• National experts are involved in 
judging proposed items in terms of 
their fit with national curricula. 

• Pilot testing is conducted on roughly 
double the amount of items that is 
actually needed for the main data 


collection, in order to determine 
which items constitute the best test. 
Psychometric analyses are con- 
ducted to check if the items fit in 
intended scales. 

• Translation verification. All items are 
originally in the English language. 
They are translated into national 
languages — which can be many, as 
in South Africa where in IEA-PIRLS 
(reading literacy) a translation into 
12 languages is needed. It is cru- 
cially important that the translation 
matches the international version as 
well as possible. The quality of the 
translations is checked by involving 
professional translators. 

• Lay-out verification. As even the lay- 
out of tests and questionnaires may 
influence the responses of the test- 
ees, the national lay-out of tests and 
questionnaires is checked at the 
international coordination centres to 
determine whether any deviations 
can be discovered. 

The question as to who is measured 
relates to the issue of populations and 
samples, which is discussed in the 
next section. 

Populations and samples 

• Population definitions 

The purpose of international assess- 
ments is to provide good national 
estimates of the indicators that have 
been defined for students, schools 
and/or teachers. The challenge is to 
define populations in such a way that 
they are comparable across countries. 
This is a complex task that cannot 
always be solved to the complete sat- 
isfaction of all participants. The IEA 
and OECD use different approaches 
for defining populations. In IEA stud- 
ies, the definitions are grade-based, 
which means that within each educa- 
tion system a particular target grade 
is chosen that corresponds to an 
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international population definition. 
Most IEA studies are focused on stu- 
dent populations at three levels in the 
education system: primary education, 
lower secondary education and upper 
secondary education. Definitions that 
were used in TIMSS2003 (for what in 
most countries constitutes the primary 
and lower secondary level) were, for 
example: ‘all students enrolled in the 
upper of the two adjacent grades that 
contained the largest proportion of 
13-year-old students at the time of 
testing and all students enrolled in the 
upper of the two adjacent grades that 
contained the largest proportion of 
9-year-olds. These correspond to the 
eighth and fourth grade in practically 
every country’ (Mullis et al. , 2004). 

The main reason for choosing a grade- 
based definition is that in IEA assess- 
ments, teachers and student data are 
linked and hence IEA is targeting data 
collection in intact classes. It should 
be noted that linkage is also possible 
when targeting individual students, but 
more complicated: for many teachers, 
their current reference point for their 
instructional activities is still an intact 
class. This is typical for the traditional 
organisation of teaching and learn- 
ing in schools. If the current reform 
trends in education (which call for 
more individual learning trajectories 
and multidisciplinary team teaching) 
are implemented on a large scale, the 
target class approach may need to be 
changed. 

Grade-based definitions do not nec- 
essarily result in comparable popula- 
tions across countries. The national 
definitions may still result in large vari- 
ability with regard to characteristics 
that impact the interpretation of the 
assessment outcomes, for instance: 
the number of years that students are 
in school may differ, while in some 


countries grade repetition occurs fre- 
quently, resulting in large age-varia- 
tion between countries. 

The approach of the OECD to defining 
student populations is different from 
the IEA: it is age-based. In PISA2003, 
the definition was ‘all students who are 
aged between 15 years 3 months and 
16 years 2 months at the time of the 
assessment, regardless of the grade 
or type of institution in which they are 
enrolled and of whether they are in full- 
time or part-time education’ (OECD, 
2004). This definition has several dis- 
advantages when it comes to the com- 
parability of populations: the number 
of years in school may differ between 
countries, and students are at differ- 
ent grade levels. A practical problem 
is the collection of data from teach- 
ers about their instructional practices 
which can be linked to the students. To 
some extent, this can be overcome by 
asking students to provide information 
about their teachers. 

An example may illustrate the prob- 
lem of differences between countries 
in terms of population characteristics. 
The results of PISA2003 showed that 
the scores of Danish students were 
moderate (roughly 36 score points 
under the top for mathematics) as 
compared to other countries, which 
resulted in some consternation among 
stakeholders in Denmark, particularly 
because the Danish education system 
is believed to be one of the best in the 
world. Certainly the expenditures on 
education are quite high (for second- 
ary education, 35 % of GDP, which 
is among the highest in the world). 
However, when comparing the charac- 
teristics of the populations of students 
from countries in PISA2003 it appears 
that Danish students were, in compari- 
son with students from other countries, 
one year less in school, because of a 
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later entry age into compulsory edu- 
cation. From TIMSS1995, Pelgrum 
and Plomp (2002) estimated that one 
year of schooling can result in score 
point differences that varied between 
13 and 44. Hence, one may wonder 
if the position of Denmark in the inter- 
national tables might be attributable 
to the deviating educational career of 
Danish students. Another complication 
of using age-based samples is that the 
logistics of data collection are much 
more complex (resulting in a heavier 
burden on NRCs and schools, causing 
higher costs and risks of higher non- 
response). 

The population definitions also have 
implications forthe definitions of teach- 
ers and school populations — that is, 
to which populations the results can 
be generalised. This is reflected in 
the way the results are presented, for 
example: 

• IEA: percentage of students by their 
school's report of teachers' involve- 
ment in professional development 
opportunities in mathematics and 
science (TIMSS2003, Exhibit 6.6.); 

• OECD: percentage of students in 
schools where the principals report 
that mathematics teachers were 
monitored in the preceding year 
through the following methods 
(Figure 5.17, PISA2003); 

• Sampling 

Once the population definitions for 
each country are settled, samples can 
be drawn. These samples need to be 
of high quality in order to warrant good 
estimates for the whole population. 
Therefore, international assessments 
apply sampling standards, which cover 
a number of aspects. 

• Accuracy: the population param- 
eters should have an accuracy for 


means of m ± 0.1s (where m’ is a 
mean estimate and ‘s’ is its esti- 
mated standard deviation) and for 
percentages: p ± 5% (where ‘p’ is a 
percentage estimate). 

• Participation rates: criteria are 

defined for participation rates that 
should be reached in order to con- 
sider a sample acceptable. 

These standards have implications, 
amongst others, for the reporting 
of the outcomes. Flags are applied 
for samples that are considered not 
too far below standards. The results 
of some countries are ‘flagged’ and 
shown ‘below the line’ which means 
that the sample quality is considered 
to be insufficient. It also happens 
that the results of some countries 
are excluded from the international 
reports, which occurred with the 
Netherlands in PISA2000. An example 
of rules for flagging from TIMSS2003 
is provided by Martin et al. (2004), see 
http://timss.bc.edu. 

Data collection and quality 
control 

The collection of data is a crucial 
phase in any ICEM. The purpose is 
that a high percentage of the sampled 
respondents answer the question- 
naires and/or tests as accurately and 
completely as possible. Any loss of 
data or inaccuracies (such as unread- 
able answers) will result in lower data 
quality and fewer possibilities for pro- 
ducing good estimates of population 
parameters. There may be many rea- 
sons why data gets lost, such as the 
following. 

• Questionnaires may not reach 
respondents, for instance because 
of failing mail services, wrong or 
unreadable addresses, sloppy 
administration in schools, etc. 
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• Answers to questions or test items 
may be unreadable or conflicting 
(e.g. more than one answer). 

• The materials are not correctly 
returned, for example because of 
wrong addresses, failing mail serv- 
ices, wrong handling at the data 
collection institute or sloppiness at 
schools (sometimes materials were 
completed but returned one year 
after data collection). 

In order to minimise data loss as much 
as possible, rigorous procedures 
are nowadays implemented in most 
ICEMs, that are all documented in 
manuals and software programs as is 
shown for instance in the TIMSS2003 
technical report (see http://timss. 
bc.edu/ for more details). 

In particular, when achievement tests 
are used, it is of crucial importance 
that the test administration takes place 
in a very controlled manner in order 
to avoid the test scores being biased 
downwards or upwards. This requires 
the following, for instance. 

• Cheating should be avoided. 

• Students need to be motivated to 
answer the test — this is particu- 
larly important because quite often 
students will perceive the test as 
low-stake as it will not have conse- 
quences for their grades in school. 

• Use of tools such as calculators 
or other aids should be standard- 
ised — this is not always possible, 
because in some countries certain 
aids are always allowed while this is 
not the case in other countries. This 
may have serious consequences 
for the interpretation of differences 
between countries. 

Nowadays many countries have to 
spend substantial budgets in order 
to guarantee the proper return of 


the instruments. This can mean that 
whole teams are busy for a consider- 
able amount of time with: 

• checking returned questionnaires 
and tests for completeness and 
readability; 

• contacting schools to get hold 
of missing materials or to clarify 
unreadable answers; 

• reminding schools by (e-)mail or 
phone to return the materials; 

• informing schools about the disas- 
trous effects when they, on second 
thoughts (after an initial agreement 
to participate), are inclined not to 
participate: sometimes the data for 
a whole country are excluded from 
the international reports. 

For planning a period for data collec- 
tion, it is important to try to avoid over- 
lap with other time-consuming and 
competing activities in school, such as 
the weeks before the school holidays, 
when everyone is busy with end-of- 
term activities, or, in some countries, 
the periods in which the final examina- 
tions are taking place. 

Data collection is one of the biggest 
budget items for national teams, 
because it is time consuming and 
requires quite high expenditures for 
materials (printing, mailing). Hence, 
one would expect that considerable 
budget reductions might be possible 
when the data are collected electroni- 
cally, via online data collection (ODC). 
ODC was not feasible for a long time, 
because respondents (schools, teach- 
ers and/or students) did not have 
access to ICT, the Internet or were not 
competent enough to use these facili- 
ties. The IEA SITES2006 was the first 
ICEM to apply ODC on a large scale. 
A feasibility test of ODC, conducted in 
two groups of respondents, randomly 
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allocated to a paper version and an 
ODC version, revealed the following. 

The analyses showed that the results 
obtained from the two modes of data col- 
lection are comparable, although there 
are some differences and issues to be 
taken into consideration. One of the most 
important issues is the level of drop-out 
in web-based questionnaires. Despite a 
higher missing item rate in web-based 
questionnaires, this method appears to 
provide a reliable data collection method 
when compared to equivalent paper- 
based questionnaires. (Brecko and 
Carstens, 2006) 

SITES2006 was a study that only 
used instruments at school and 
teacher level. The sample sizes for 
these categories of respondents are 
usually relatively small in ICEMs and, 
hence, the efficiency profit is much 
less than when ODC can also be used 
for students in those assessments that 
administer tests and/or questionnaires 
to students. Feasibility tests of ODC 
for large-scale student assessments 
still need to be tried out. 

Additional advantages of applying 
ODC in the future might be: 

• negligible costs for data entry (see 
next section); 

• tailored testing; 

• performance testing, such as testing 
via simulations, practical laboratory 
skills, communication competen- 
cies, etc.; 

• possibility to provide more direct 
and timely feedback to respondents 
(which may increase willingness to 
participate); 

• more continuous and periodic moni- 
toring for large samples of schools 
and possibilities for school self 
evaluation; 

• more possibilities for diagnosis. 


More research is needed to investigate 
and try out these possibilities. This 
requires staging and cooperation at 
the international level, involving differ- 
ent partners that are still working inde- 
pendently (such as national and inter- 
national assessment organisations). 

Data entry and file building 
The purpose of data entry is to enter 
the answers from respondents accu- 
rately in data files. This should prefer- 
ably be done by highly qualified key 
punchers. But, as this is a human 
activity, failures are possible (this also 
holds for optical mark reading, where 
human interventions to solve ambigu- 
ous responses are needed). Such fail- 
ures may, if undiscovered, have a huge 
impact on the statistical estimates. 

There are several tools used to dimin- 
ish data-entry failures: 

• use of data-entry programs that con- 
tain immediate checks when data 
are entered, such as: 

— valid codes for categorical 
variables, 

— valid ranges (e.g. for the school 
sizes, number of computers, 
etc.); 

• data checks that consist of, for 
example: 

— ID checks: every identification 
number for respondents in the 
data files should be unique, 

— linkage checks: every student 
should be linkable to a unique 
teacher and unique school, 

— inconsistency checks: the 

answer to a filter question should 
not be in conflict with subsequent 
questions; 

• data analytical checks in the national 
centre as well as the international 
data management centre, such as: 

— distribution of answers should 
be plausible when compared 
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with other statistical sources in 
a country, e.g. from other inves- 
tigations or national census 
statistics, 

— the ‘behaviour’ of variables 
should be plausible, e.g. an unu- 
sually high correlation between 
two variables in one or a few 
countries may be suspicious and 
could point to errors in the data, 

— examining differential item-func- 
tioning, e.g. relative high p-val- 
ues in one country as compared 
with other countries could poten- 
tially (but not necessarily) point 
to flaws in the translation of test 
items. 

When potential problems are discov- 
ered in the data, it is often necessary to 
go back to the returned questionnaires 
and/or tests to find out what was actu- 
ally entered by the respondents. 

It goes without saying that by applying 
ODC, many of the problems that result 
from human failures during data entry 
can be avoided. However, ODC is not 
a panacea for getting error-free data, 
because: 

• respondents may accidentally hit 
wrong keys - it is not known whether 
this is maybe more likely than acci- 
dentally hitting a wrong answer in a 
printed questionnaire (e.g. for a filter 
question); 

• for open questions requiring the 
specification of a number (e.g. 
number of students in school), 
respondents may accidentally write 
a wrong number. 

Part of the procedure for checking and 
cleaning data is: 

• inspection of national univariate stat- 
istics by each participant; 

• inspection of international univari- 
ates by national research coordin- 


ators (NRCs) and the international 
coordinating centre (ICC) — quite 
often suspicious statistics are dis- 
covered by comparing univariate 
statistics and sometime even at 
a late stage by inspecting the out- 
comes of the analyses, for instance 
as a result of an undiscovered trans- 
lation error (e.g. ‘don't mind' trans- 
lated as ‘don’t like’ in SITESM1). 

Errors in the data are not always caused 
by data entry failures. They may also 
result from printing errors or national 
adaptations in questionnaires. 

Due to all these checking and valida- 
tion steps, it can take some time to 
produce international data sets that 
are ready for further processing and 
analysis. However, it is usually not 
until the first official report is published 
(about a year after data collection) 
that the data sets are considered to 
be in their final shape and ready for 
public access. This is because, as 
mentioned above, even at a late stage 
errors in the data can be detected. 

The purpose of data processing is to 
produce statistics that were envisaged 
when conceptualising and designing 
the ICEM. These statistics may be: 

• univariate and based on one vari- 
able (e.g. a percentage of students 
having a computer at home) or com- 
posed on a set of variables (e.g. a 
mean number of possession from a 
set of 1 0 in students’ homes); 

• bivariate, for instance breakdowns 
of such test scores for boys and girls 
or correlations, e.g. between score 
on a like-math-scale and the math- 
achievement score; 

• multivariate, e.g. structural models 
that are fitted on the data. 
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A persistent problem in data process- 
ing is how to handle missing data, such 
as when respondents (intentionally or 
accidentally) have not answered a 
question. For example, the following 
cases may occur. 

• Missing should be interpreted as 0. 
Although respondents are explicitly 
instructed to write a zero if that is the 
answer to particular open questions, 
this does not always happen, and 
bias can be introduced in the statis- 
tical estimates. 

• Missing to be interpreted as neu- 
tral, such as when response scales 
are used without a neutral answer 
category. 

• Missing by design, such as when 
matrix sampling ( 2 ) is used. 

Missing data that result from design 
are often replaced by imputed values. 
Mostimputationstake place via regres- 
sion analyses in which a large number 
of variables are used to predict scores 
on the variable that contains missing 
codes. Once the regression weights 
are known, these are used to ‘predict’ 
the score for the missing answers. 

How to handle other missing data 
requires a close inspection ofthe data, 
because, as argued above, hypoth- 
eses need to be generated about what 
missing may mean. 

Avery important step in data process- 
ing is the calculation of appropriate 
standard errors for the statistics that 
are produced. A standard error is an 
estimate ofthe sampling inaccuracy. It 
is used to describe the so-called confi- 
dence interval for statistical estimates 
of population parameters. For simple 


( 2 ) Matrix sampling means that a sample of 
questions is administered to a (sub)sample of 
respondents. 


random samples, this can be simply 
calculated by dividing the standard 
deviation of a statistic by the square 
root of the number of cases. As was 
explained in the section about popu- 
lations and sampling, ICEMs are not 
normally based on random samples of 
students, but are rather so-called clus- 
ter-samples: first schools are selected 
and then students within schools. 
As the units in these clusters usu- 
ally resemble each other, the effect 
of this approach is that less accurate 
estimates can be made. Hence, using 
statistical tests from standard software 
such as SPSS is not correct and there- 
fore appropriate dedicated procedures 
need to be developed for each ICEM 
separately. 

Data analysis 

The purpose of data analysis in gen- 
eral is to find answers to several types 
of questions, such as the following. 

• ‘Why’ questions, e.g. ‘Why are the 
achievement scores in certain coun- 
tries low?’, ‘Why are the scores 
on emerging-practice indicators in 
some countries much higher than in 
other countries?’. 

• Questions about hypothesised rela- 
tionships: e.g. ‘Is the availability of 
ICT related to the extent that emerg- 
ing pedagogical practices exist in 
schools? 

• Exploratory questions: ‘Which 

school factors are associated with 
the existence of emerging pedagog- 
ical practices? 

In the current international descrip- 
tive reports, variables can be found 
that could be of interest for further 
analysing the data, e.g. breakdowns 
of achievement scores by different 
groups of students, those having com- 
puters at home, low, medium and high 


55 


Chapter II — State of the art 


social welfare index, etc. The current 
reports, particularly the PISA reports, 
also contain initial results of more in- 
depth analyses. 

However, these analyses do not offer 
more than a first approach to the analy- 
sis of the data. For a comprehensive 
analysis, the ‘behaviour' of a large set 
of variables needs to be taken into 
account, which is often done by fitting 
models on the data (confirmatory, that 
is, based on an a priori hypothesised 
structure; or exploratory and aimed at 
generating a posteriori hypotheses, 
which is more common: by trying out 
many different models and by deter- 
mining which model fits the best). 
Examples of statistical programs for 
modelling are LISREL and AMOS 
(part of the SPSS package). As the 
data often have a multi-level character 
(school-, teacher- and student-level), 
so called hierarchical linear modelling 
(HLM) programs are also used. Finding 
appropriate models that fit the data well 
is a time-consuming process, which 
often takes place after the first descrip- 
tive ICEM reports have been published. 
It should, however, be noted that the 
OECD included quite a lot of multivari- 
ate analyses in the PISA reports. 

Sometimes special issues of journals 
or dedicated books are devoted to 
secondary analyses of the assess- 
ment data (e.g. Robitaille and Beaton, 
2002). However there is a lack of up- 
to-date meta-analyses, showing which 
analyses have been done over the 
years and which results have been 
reported. Such an activity is important, 
among other reasons because it is not 
yet very well understood why some 
variables are highly intercorrelated 
in some countries but not in others. 
Also, as mentioned before, quite often 
constraints of studies do not allow for 
enough variables covering the a poste- 


riori research questions. This in itself is 
not a fundamental problem, but rather 
the lack of a coherent and long-term 
research agenda is, or in the words of 
Martin et al. (2004): ‘more work needs 
to be done to identify the most fruitful 
variables to capture the dynamic proc- 
esses that take place within schools 
and to understand how national and 
cultural contexts interact with other 
factors to influence how education is 
transmitted and received’. 

Reporting 

As argued earlier in this chapter, an 
important step in any ICEM is the valu- 
ation of the results. ICEM reports offer 
a rich variety of statistics that can help 
the participants to judge the results 
for their country. In ICEMs this is usu- 
ally a relative judgment, that is, coun- 
try statistics are valued on the basis 
of comparisons with other countries. 
A danger in interpreting the statistics 
may be that too much of an atomistic 
approach is used (focusing on one or 
a few subject areas) rather than trying 
to value an education system from a 
holistic perspective. 

However, it can be observed that once 
the final report has been released, 
absolute judgments also enter the 
scene, e.g. some people claiming that 
despite the high score of a country in 
fact the quality of maths achievement 
is very low. This happened recently 
in the Netherlands, when a group of 
researchers from the Freudenthal 
Institute for Science and Mathematics 
Education concluded that, despite the 
high international ranking, the level 
of achievement in the PISA tests was 
very low. 

Secondary analyses 

ICEMS result in huge data sets 

(50 countries with on average 
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5 000 students per country is not 
uncommon) that are nowadays easily 
accessible for several purposes. The 
background documents on design 
and methodological issues (sampling, 
technical standards, psychometrics) 
also reflect how researchers in the field 
apply theoretical insights from educa- 
tional methodology. These data can be 
of value for examining and illustrating 
several methodological topics and for 
conducting substantive research after 
the data have been archived, includ- 
ing the following. 

• Conceptualisation (concepts and 
indicators): every interested person 
can have access to instruments, con- 
ceptual frameworks and data and, 
hence, can be involved in reflecting 
about the choices that were made in 
particular assessments. 

• Questionnaire development: by criti- 
cally examining questionnaires that 
have been used in international 
assessments, forming hypotheses 
aboutthe strong and weak points and 
analysing the data to find evidence 
for these hypotheses, much can be 
learned about issues that concern 
questionnaire development. 

• Sampling: several issues are worth 
examining and discovering in the 
international data files. 

— Is the accuracy of the population 
estimates comparable to theo- 
retical expectations? 

— Do education systems where 
streaming occurs have higher 
intra-class correlations than sys- 
tems where this is not the case? 

• Data collection: international com- 
parative assessment projects over 
the past 30 years have developed a 
whole set of tips and tricks for col- 
lecting high-quality data from large 
samples of students, teachers and 
schools in a country. 


• Data analysis: international com- 
parative data sets nowadays offer a 
wealth of opportunities to investigate 
how certain measures behave under 
different circumstances. Questions 
include: ‘do attitude measures from 
Japanese and UK data show the 
same underlying dimensions?’. 

• Substantive questions: international 
comparative assessments typically 
cover a broad range of topics. For 
instance, the tests for measuring 
student achievement may con- 
tain hundreds of questions cover- 
ing a large part of the mathematics 
domain. Detailed examination of 
these items may reveal much more 
than the overall test statistics which 
are published in the international 
reports. 

Whereas in earlier days the use of 
international databases was compli- 
cated (often the data files which were 
stored on tape did not even fit on hard 
disks of mainframe computers) nowa- 
days anyone with a relatively simple 
laptop can download the data bases 
and conduct analyses. Such analyses 
usually require an in-depth under- 
standing of technical details, such as: 

• which sampling weights are avail- 
able in the data files and how these 
should be used: 

• how to calculate standard errors in a 
correct way, taking into account the 
sampling design of the studies; 

• how to apply the so-called plausible 
values that are stored in the files. 

However, when carefully studying 
the technical documentation and 
user guides that are available for the 
ICEMs, secondary analyses are pos- 
sible for almost everyone with some 
affinity for statistics. 
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PISA data can be explored online ( 3 ). 

A tool that may be useful in doing sec- 
ondary analyses on IEA data is the 
IEA International Database Analyzer 
(IEA IDB Analyzer), a plug-in for SPSS 
that helps to correctly handle data and 
which can be found at http://www.iea. 
nl/iea_software.html. 

Additional methodological 
considerations for monitoring 
ICT in education 

In addition to the general methodologi- 
cal issues that were (not exhaustively) 
reviewed above, when monitoring ICT 
the following additional issues should 
be considered. 

• Terminology: In questionnaires used 
for collecting data with regard to ICT 
indicators, the term ‘computer’ is 
often used as a stand-in for the more 
general term ‘ICT’. Given the techno- 
logical developments in recent years 
it is questionable whether ‘computer’ 
adequately covers the current tech- 
nology options. For example, when 
students in primary education are 
asked whether they use computers 
during lessons and where they only 
use interactive whiteboards, one 
may wonder whether they recognise 
this device as a computer. In this 
case, the use of the word ‘computer’ 
may lead to downwards biased esti- 
mates of ICT use during lessons at 
school. The same problem applies 
to questions like ‘Do you use a com- 
puter for accessing the Internet?’ It 
is quite likely that students who use 
mobile phones for accessing the 
Internet would answer ‘no’ to such 
question. Maybe this could be an 
explanation for the extremely low 


( 3 ) For PISA 2003: http://pisaweb.acer.edu.au/ 
oecd_2003/oecd_pisa_data_s2.php. 


use of the Internet as reported by 
Japanese students (see Chapter 4). 

• Self-ratings: Quite often in inter- 
national as well as national ICT 
monitors, instead of using objective 
standardised tests, students and / 
or teachers are asked to rate their 
own ICT competencies. Although 
such measures may be fine as indi- 
cators of self-confidence, they are 
often used as proxies for real com- 
petences. Such use is unwarranted, 
as self-ratings are prone to bias 
(Stromsheim, 2002; Ross, 2006). 

• Teacher perceptions: Some assess- 
ments in the past included percep- 
tions of teachers regarding the 
impact of ICT on, for instance, moti- 
vation and skills of students. The 
validity of such measures is highly 
questionable and the ratings are 
prone to wishful thinking. Hence, in 
future assessments, such measures 
should only be used as an indicator 
of teachers’ attitudes towards ICT. 

Summary and conclusions 

In the previous sections, a number of 
key terms were introduced that play a 
role in monitoring. A key term for this 
study is the word ‘indicator’, which 
does not have an unequivocal defini- 
tion. Literally it means ‘an indication of 
something that is not directly observa- 
ble’. Indicators may be categorised in 
terms of global descriptions of rather 
broad areas, or more concrete defini- 
tions. For instance, a broad area con- 
cerns ICT infrastructure in education. 
Many different indicator definitions 
may be distinguished within this area, 
such as ‘Quantity of available PCs in 
schools’. Such definitions are guiding 
the development of instruments to col- 
lect data, which consist of response- 
codes delivered by respondents and 
stored in data files. Once these data 
are available, several statistics may 
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be calculated in order to provide a 
quantitative estimate for the targeted 
indicator. Hence these will be called 
‘indicator statistics' in this study. For 
instance, examples of statistics for 
the indicator definition just mentioned 
might be: mean number of comput- 
ers per school in each country or the 
median number of computers. Another 
relevant statistic might be the mean 
number of students per available com- 
puter in a country’s education system. 

It is conceivable that data are avail- 
able for which no indicator statistics 
exist, which can quite often happen 
in international comparative assess- 
ments due to space limitations in the 
final reports. Indicator areas may also 
exist for which no indicator definitions 


are available. Hence, these termino- 
logical distinctions were relevant for 
the purpose of our study. 

The distinction between primary and 
secondary indicators was also intro- 
duced and the problem was men- 
tioned of defining appropriate second- 
ary indicators before the questions for 
the phase of diagnosis are generated. 
It was pointed out that in order to avoid 
undesirable impacts on educational 
decision-making, holistic monitoring is 
needed. It was also argued that multi- 
level monitoring may be an important 
option for the future. Several potential 
advantages of online data collection 
were mentioned that may play a role 
in further discussions about a future 
EU ICT monitor. 
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Executive summary 

infoDev maintains a series of knowledge maps that attempt to document what is 
known — and what is not known — about ICT use in education. These knowledge 
maps reveal that, despite a decade of large investment in ICT to benefit educa- 
tion in OECD countries, and increasing use of ICT in education in developing 
countries, important gaps remain in our knowledge. In addition, there appears 
to be a dearth of useful resources attempting to translate what is known to work 
and not work in this field for policymakers and donor staff working on education 
issues in developing countries, especially those issues related to ‘education for all’ 
and other education-related millennium development goals. A lack of reliable data 
related to the impact of ICT on learning and achievement in developing countries, 
as well as a lack of useful indicators and methodologies to measure such impact, 
hampers policy guidance in this area. A mismatch also exists between methods 
used to measure the effects of ICT use in education in developing countries, and 
type of learning styles and practices that the introduction of ICT is meant to pro- 
mote, or at least facilitate. 

Despite a lack of reliable impact evidence, recent infoDev surveys of World Bank 
support for ICT components in projects in its education portfolio, and country-level 
surveys sponsored by infoDev of ICT use in education in Africa and the Carib- 
bean, document tremendous growth in the use of — and demand for — ICT in the 
education sector. This mismatch between weak evidence and growing use raises 
many questions about the nature of ICT-related investments in the education sec- 
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What do we know 
about the effective 
uses of information 
and communication 
technologies in education 
in developing countries? 

/'ntoDev maintains a series of ‘knowl- 
edge maps’ outlining what is known 
— and what is not — about the use of 
information and communication tech- 
nologies (ICT) in education. These 
knowledge maps reveal that, despite a 
decade of heavy investment in ICT to 
benefit education in OECD countries, 
and increasing use of ICT in educa- 
tion in developing countries, signifi- 
cant gaps remain in our knowledge. In 
addition, there appears to be a dearth 
of useful resources for policymakers 
and donor staff working on education 
issues in developing countries, iden- 
tifying what is known to work — and 
not to work — in this field, especially 
in support of ‘education for all’ (EFA) 
and other education-related millen- 
nium development goals (MDGs) (see 
Trucano, 2005). 

The knowledge maps, which are used 
to help guide discussions between 
donors and governments exploring 
the use of ICT in the education sector, 
investigate 1 0 topics (impact of ICT on 
learning and achievement, monitoring 
and evaluation, equity issues, costs, 
current projects and practices, spe- 
cific ICT tools, teaching and ICTs, con- 
tent and curriculum, policy issues, and 
school-level issues). The key findings 
are divided into four major themes. 

Key findings: Impact 
• The impact of ICT use on learning 
outcomes is unclear, and open to 
much debate. 


• Widely accepted, standard method- 
ologies and indicators to assess the 
impact of ICT in education do not 
exist. 

• Adisconnection is apparent between 
the rationales most often presented 
to advance the use of ICT in educa- 
tion (to introduce new teaching and 
learning practices and to foster ‘21 st 
century thinking and learning skills’) 
and their actual implementation 
(predominantly for use in computer 
literacy and dissemination of learn- 
ing materials). 

Key findings: Costs 

• Very little useful data exists on the 
cost of ICT in education initiatives, 
especially related to total cost of 
ownership and guidance on how to 
conduct cost assessments. 

Key findings: Current 
implementation of ICT in 
education 

• Interest in and use of ICT in educa- 
tion appears to be growing, even in 
the most challenging environments 
in developing countries. 

Key findings: Policy lessons 
learned and best practices 

• Best practices and lessons learned 
are emerging in a number of areas, 
but, with few exceptions, they have 
not been widely disseminated nor 
packaged into formats easily acces- 
sible to policymakers in developing 
countries, and have not been explic- 
itly examined in the context of the 
education-related MDGs. 

While much of the rhetoric about (and 
rationale for) using ICT in education 
has focused on the potential for chang- 
ing the teaching-learning paradigm, in 
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practice ICTs are most often used in 
education in less developed countries 
(LDCs) to support existing teaching 
and learning practices with new (and, 
it should be noted, often quite expen- 
sive) tools. While impact on student 
achievement is still a matter of reason- 
able debate, a consensus seems to 
have formed that the introduction and 
use of ICT in education can help pro- 
mote and enable educational reform, 
and that ICT is a useful tool to both 
motivate learning and promote greater 
efficiencies in education systems and 
practices. 

Surveys of ICT use in 
education in developing 
countries: what is actually 
happening? 

Research teams supported by infoDev 
and coordinated bytheCommonwealth 
of Learning (COL) and others are 
seeking to document the major devel- 
opments in each country in Africa (see 
Farrell et al„ 2007a, 2007b, 2007c) 
and the Caribbean (see Gaible, 2007) 
related to technology use in education 
in order to create the first consolidated 
look at this fast-changing sector in 
these regions and provide preliminary 
answers to three broad questions. 

• How is ICT currently being used in 
the education sector, and what are 
the strategies and policies related to 
this use? 

• What are the common challenges 
and constraints faced by countries 
in this area? 

• What is actually happening on the 
ground, and to what extent are 
donors involved? 

infoDev and its partners hope that 
release of the results from these 
surveys, and related monitoring and 


evaluation studies of key initiatives 
like NEPAD e-Schools, is a first step 
in a larger, ongoing, systematic and 
coordinated initiative to track develop- 
ments in technology use in the educa- 
tion sector to help inform a wide vari- 
ety of stakeholders interested in the 
topic as they seek solutions to larger, 
more fundamental educational and 
development challenges in the years 
ahead. 

Key findings 

• ICT use In schools in Africa and the 
Caribbean is growing rapidly (from 
an admittedly low base). 

This growth is largely the result of 
‘bottom up’ initiatives, often facili- 
tated by civil society organisations. 
Barriers to use include high costs 
(especially of connectivity), poor 
infrastructure, insufficient human 
resource capacity, high costs, a 
variety of disincentives for use and 
inadequate or insufficient policy 
frameworks. 

• The process of adoption and diffu- 
sion of ICT in education in Africa is 
in transition and widely variable. 

A marked shift seems to be emerg- 
ing from a decade of experimenta- 
tion in the form of donor-supported, 
NGO-led, small-scale pilot projects 
towards a new phase of systemic 
integration informed by national 
government policies and multi- 
stakeholder-led implementation pro- 
cesses. This shift from projects to 
policies, and the more systematic 
development that that implies, would 
not be possible without the growing 
commitment to ICT in education 
on the part of government leaders 
across the continent’ (Farrell/lsaacs, 
2007). 

• ICT use in education in the 
Caribbean, and the context of its 
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use , varies only within a limited 
range. 

ICT use in schools in the region is 
primarily centred on basic ICT liter- 
acy instruction and computer use. 

Planning for ICT use in 
education in developing 
countries: a way forward 
for policymakers 

As an aid to education policymakers 
in developing countries under tre- 
mendous pressure — from parents, 
vendors, business, technology advo- 
cates, etc. — to provide schools with 
a variety of ICT, infoDev, Unesco and 
others partners have developed and 
utilised an ICT-in-education toolkit 
as part of policy consultations in 
26 countries (see Haddad, 2007). 
Feedback from toolkit users consist- 
ently states that provisioning ICT for 
use in schools, no matter how hard 
and expensive initially, is the easiest 
and cheapest element in a series of 
policy choices that ultimately could 
make ICT use sustainable and/or ben- 
eficial for learners. Indeed, the appro- 
priate and effective integration of ICT 
in schools to impact teaching and 
learning practices is much more com- 
plicated. The proliferation of ICT use 
outside the school — especially the 
growing use of mobile phones — has 
yet to impact in any meaningful way on 
the use of ICT within formal education 
systems. To help guide policy choices 
around technology use and choice in 
education in developing countries, a 
more robust set of shared indicators 
and evaluation methodologies must 
be developed and tested in real- 
world circumstances. As discussed in 
infoDev's Monitoring and evaluation of 
ICT in education projects: a handbook 
for developing countries, ‘evidence 
to date suggests that policymakers 


and project leaders should think in 
terms of combinations of input factors 
that can work together to influence 
impact. Coordinating the introduction 
of computers with national policies 
and programmes related to changes 
in curriculum, pedagogy, assessment 
and teacher training is more likely to 
result in greater learning and other 
outcomes’ (Wagner, 2005). 

The process of integrating ICT into 
educational systems and activities can 
be (and typically is) arbitrary, ad hoc 
and disjointed, as evidenced through 
recent infoDev surveys of ICT use in 
education in the 75 developing coun- 
tries (Farrell et al. , 2007a, 2007b, 
2007c, Trucano, 2007). Such adhoc- 
racy often results in ineffective, unsus- 
tainable and wasteful investments. On 
the other hand, a comprehensive set 
of analytical, diagnostic and planning 
tools, such as those promoted through 
the ICT in education toolkit, can 'force 
a certain discipline on the process. 
The use of tools does not make policy 
formulation ‘scientific’ and ‘rational’. 
Nor will it replace the political/organi- 
sational nature of policy formulation’ 
(Haddad, 2007). 

That said, it is clear that current tools 
available to help aid policymakers 
make informed decisions about 
technology choices for schools are 
quite primitive. Reasonable minds can 
argue over what is meant by ‘impact’ 
and ‘performance’, but substituting 
belief for scientific inquiry does not 
seem to be a particularly responsible 
course of action. The power of ICT 
as an enabler of change — for good, 
as well as for bad — is undeniable. 
However, the use of ICT in education 
in many developing countries, 
especially the ‘poorest of the poor’, 
is associated with high cost and 
potential failure. Simply wishing away 
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the existing local political economy of 
the way technology is implemented 
and supported in schools does not 
mean that it actually goes away. With 
more rigorous analysis and evidence 
of impact, and better decision tools, 


developing country policymakers — 
and their partners in the international 
community — can make wiser and 
more sustainable choices in deploying 
ICT to enhance access to, and quality 
of, education at all levels. 
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A framework for understanding and 
evaluating the impact of information and 
communication technologies in education 

Katerina Kikis, Friedrich Scheuermann and Ernesto Villalba 

During the last decades, considerable resources have been invested in hardware, 
software, connections, training and support actions under the scope of improving 
the quality of teaching and learning. A major tenet of the policies that supported 
the introduction of information and communication technologies (ICT) in educa- 
tion was that they can become catalysts for change. Undoubtedly, some countries 
have made considerable progress in bringing networked ICT into education and 
made it possible for teachers and learners to use them on a daily basis. In many 
other cases, however, implementation policies have not been a consequence of 
systematic analysis and reflection. /As a consequence, we still know little about 
the impact and effectiveness of ICT in education. To close this gap, the Center 
for Research on Lifelong Learning based on benchmarks and Indicators (CRELL) 
established a research project on measuring ICT performance and effectiveness 
in education. The project explores the effects of ICT on learning outcomes aiming 
at stimulating debate on educational policy needs. This paper presents the first 
step in the process. It presents a conceptual framework to guide the analysis for 
orienting work activity towards the study of ICT effectiveness. 


Integrating ICT in education 

The integration of ICT in education 
is affecting educational systems in 
multiple ways. Likewise, ICT use in 
education influences the private life 
of all educational actors in the sense 
that these are engaged in innovative 
practices which require new meth- 
odologies, techniques and attitudes. 
Most studies carried out, however, do 
not provide clear information about 
multifaceted effects and impact of 
"on the learner and learning. There 
areTsJill unanswered questions about 
the impact of technology in the short 
and long terms on learning and how 
it has affected simple^ and complex 
learning tasks. In turn, this has impor- 


tant consequences in the articulation 
of educational policies. The identified 
gap in assessing the impacts of ICT is 
especially unsatisfying for policymak- 
ing stakeholders who aim at defin- 
ing evidence-based strategies and 
regulatory measures for effective ICT 
implementation and efficient use of 
resources. 

Emerging technologies (e.g. smart- 
boards, mobile devices) stimulate the 
change in contextual conditions for 
learning. Computer equipment and 
software are becoming increasingly 
available inside educational estab- 
lishments as well as in private house- 
holds — not only for school-related 
activities of young people, but also for 
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learning at all stages in life. Instruc- 
tional practices are changing due to 
new possibilities to access and share 
information, new roles and peda- 
gogical paradigms. Furthermore, we 
observe new ways of learning in the 
context of new educational software 
applications and tools provided, dig- 
ital resources available, etc. (see, for 
example, Redecker, 2009). This justi- 
fies once more the need to study the 
effects of ICT at different levels and 
to examine implications for the indi- 
vidual and society. More insights into 
the multifaceted effects are needed 
to enable us to conduct cost-benefit 
studies in an appropriate manner 
and to react to necessary changes by 
updating national curricula, design- 
ing teacher training programmes 
and revising adequate school and 
classroom implementation, keeping 
in mind that ICT is often a catalyst 
for change but does not itself deter- 
mine the direction of change. There 
is a lack of comprehensive studies 
of the complex interactions between 
various types of ICT implementation 
and the effects of other factors such 
as school-based interventions, socio- 
economic status and expenditure. It 
appears that, firstly, we are in need of 
instruments which will allow assess- 
ing and monitoring the state of use 
and changes affected. Secondly, we 
need to identify the various sources 
and gaps in a systematic manner in 
order to determine data available and 
desired. There are a number of ambi- 
tious initiatives to explore the scope 
of influencing factors already car- 
ried out (see, for example, Ramboll 
Management, 2006; Underwood etal., 
2007). They provide a good basis for 
going one step further and designing 
a systematic approach to identify the 
use of ICT and its effects on all differ- 
ent levels and stages concerned. 


In many cases, in the context of school 
education, the massiveness of gov- 
ernment top-down ICT-related pro- 
grammes and reforms implied that poli- 
cymakers were expecting schools to 
change sooner rather than later. Unlike 
books or blackboards, digital technolo- 
gies tend to age and even become 
unusable within just a few years. 
Furthermore, technology changes very 
fast and even if older technology is 
still usable it can be incompatible with 
new digital products and services or 
be unsuitable for their full exploitation. 
Overall, this top-down approach has 
had its own risks because the heavy 
investments could pay back only if 
schools were ‘ready enough’ to start 
immediately using ICT in productive 
ways. The massiveness of the pro- 
grammes and reforms introduced also 
implied that the changes anticipated 
were envisaged to take place not just in 
some or even in the majority, but in all 
schools within a system. The reformers 
probably pushed ahead because they 
wanted to minimise the risk of creat- 
ing inequalities among schools which 
make heavy use of ICT and those that, 
for one reason or another, do not. The 
scenario, however, that assumed that 
all schools would start using ICT in 
productive ways as soon as the teach- 
ers and the pupils put their hands on it 
was not very realistic. What was more 
plausible was that the top-down pro- 
grammes and reforms would gradu- 
ally help more and more teachers and 
pupils alter their teaching and learning 
practices. According to this scenario, 
the early adopters who used ICT prior 
to the implementation of massive top- 
down programmes and reforms will 
soon be joined by an early majority, and 
the sceptics, what Rogers (1 995) called 
the ‘late majority’, will eventually follow 
them. As teachers and pupils convert 
from being non-users to regular users 
of ICT for teaching and learning, they in 
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parallel learn how to use them in opti- 
mal ways, i.e. as they learn something 
new, they learn new ways to learn. In 
other words, according to this scenario, 
ICT will ‘penetrate’ and change schools 
in successive stages. 

Indicators for monitoring 
the integration of ICT 
in education 

Such ‘outside-inside mentality’ is also 
evident in widespread approaches 
to the evaluation of the integration 
and impact of ICT in school life. At 
national and cross-national level, a 
widespread approach to evaluation 
is through indicators. Indicators, as 
defined by Unesco (2003), are meas- 
uring devices to assess or evaluate 
materials, methods, an intervention, a 
programme or a project on the basis 
of adopted assumptions on what is rel- 
evant. Many countries worldwide have 
adopted quantitative and qualitative 
indicators of the degree of integration 
of ICT into schools and some of them 
have even established annual surveys 
to monitor progress in this area. 

Input indicators are the most widely 
used type of indicators, something 
that reflects the priorities of national 
policies, which commonly focus first 
on building a minimum level of ‘frame- 
work conditions' in schools. The great- 
est emphasis has been placed on 
input indicators regarding national 
policies and the regulatory frame- 
works, expenditure, teacher training, 
the inclusion of ICT in school curricula, 
ICT infrastructure in schools and the 
access of ICT equipment by teachers 
and pupils at home. As ICT gradually 
becomes an integral part of schools 
and elsewhere, and many teachers 
receive training in ICT, the interest 
has shifted towards issues concerning 


how teachers and pupils actually use 
ICT (utilisation indicators), what the 
outcomes are of their use (outcome 
indicators), and, more recently, what 
the impact is of their use on school 
learning (learning impact indicators). 
Utilisation indicators often measure 
how often teachers and students use 
ICT for school teaching and learning, 
what they use and for what purposes 
(for example, what kind of software 
they use for subject teaching and learn- 
ing), and how they use it (for exam- 
ple, whole-classroom teaching, group 
work, individual work, etc.). Outcome 
indicators often focus on the attitudes 
of teachers and pupils towards ICT, 
and their confidence and skills in using 
ICT. They also start to focus on wider 
‘strategic’ practices such as the use 
of ICT for lifelong learning and pro- 
fessional development, and assess- 
ment of actual ICT skills is starting 
to be developed in some areas. It is, 
however, much less common to use 
indicators to measure the impact of 
the use of ICT on pupils’ attainment in 
core curriculum subjects. 

The development and use of indica- 
tors is popular among policymak- 
ers because they provide them with 
a wealth of easy-to-use information. 
However, it is important to bear in 
mind that the use of indicators has 
its limitations: generally, indicators 
provide support to assess a current 
state, but usually do not cover other 
important issues, such as reasons 
for not using ICT; mental effects on 
learner and learning, etc. Moreover, 
comparative surveys typically only 
provide a snapshot of a given situa- 
tion at a very specific moment in time. 
Furthermore, the choice of mainly 
‘input’ indicators is often driven by 
political priorities and the philosophy 
and concerns of the bodies, often 
government supported, issuing such 
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studies. Therefore, the indicators tend 
to focus on areas where there has been 
a recent policy initiative and they tend 
to ignore other areas which, although 
highly relevant, are not included in the 
current policy agenda or may reveal 
disturbing policy failures. For ex- 
ample, the use of the ratio between 
pupils and computers and the ratio 
between teachers and computers as 
input indicators draws a picture which 
may be quite different from the pic- 
ture which would result if the teacher: 
pupil ratio was also included as a third 
indicator. From a wider perspective, 
the indicators approach often reflects 
the wider top-down, outside-inside 
mentality that was adopted through 
the implementation of massive pro- 
grammes and reforms. In a way, it is 
a consistent part of a wider top-down 
policymaking culture which assumes 
that the starting points for generating 
school change are the actions of poli- 
cymakers (Kollias and Kikis, 2005). 

From a European perspective, the 
development and use of indicators is 
highly relevant, especially forthe devel- 
opment of monitoring policies estab- 
lished by the European Union. The 
Lisbon strategy set up the open method 
of coordination (OMC) in education 
and training (among other fields). This 
implies that Member States agreed to be 
monitored in a series of issues to allow 
for mutual policy learning. In 2002, five 
benchmarks were established as the 
average level to achieve by 2010 and 
several indicators were proposed for 
monitoring purposes. In addition, the 
recent emphasis on evidence-based 
policies in education (see European 
Commission, 2007a) (') also provides a 


(') European Commission (2007). 'Towards 
more knowledge-based policy and practice 
in education and training’. SEC (2007) 1098. 
Luxembourg: Office for Official Publications of 
the European Communities. 


strong policy support for the creation of 
monitoring tools in education. In 2007, 
the Commission published the coher- 
ent framework of indicators (European 
Commission, 2007b). This communica- 
tion established 16 indicators that were 
adopted by the European Council and 
can be used to monitor Member States 
in the achievement of the Lisbon goals 
in education and training, one of which 
is ‘ICT skills’. In the current state, there 
is a necessity to place this indicator 
within a wider context of ICT use and 
integration. Likewise, other European 
programmes, such as i2010, aim at 
promoting the positive contribution of 
ICT in the economy, society and qual- 
ity of life. There is a need to have a 
framework that will allow evaluating 
the impact of ICT for this purpose, par- 
ticularly its contribution in educational 
settings. 

Existing comparative data 

The OMC, as well as the trend of 
proposing knowledge-based policies, 
requires reliable data and information 
for policymakers to enable the moni- 
toring of policies. Data needs to be 
comparable in order to allow for mutual 
learning between countries. In prin- 
ciple, it is possible to group potential 
sources and instruments for assess- 
ing the ICT effect at a comparative 
level into three different categories: 

• data collected by international 
bodies (Eurostat, World Bank, 
Unesco, OECD); 

international surveys, (such as PISA, 
TIMSS, PIRLS, SITES, TALIS); 

• thematic studies (e.g. ‘Study of the 
impact of technology in primary 
schools' (STEPS) 2009, carried out 
by European Schoolnetand Empirica 
forthe European Commission). 
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Additionally, national experiences and 
studies are a good source of informa- 
tion. These, however, do not allow 
for comparison across countries in a 
straightforward manner. These consti- 
tute ‘case studies’ and could be used 
as ‘lessons learned'. For the present 
paper, the focus remains on the com- 
parative, international sources of 
information. 

The data compiled by international 
bodies might be instrumental in pro- 
viding the context for the effects of 
ICT in education. Any effect has to 
necessarily be related to the context 
where it has appeared. In this regard, 
several international bodies collect 
information on ICT infrastructure. 
The OECD, for example, publishes 
the Communications outlook and the 
Information technologies outlook every 
two years. These two publications pro- 
vide an overview of the situation in the 
telecom market. They contain plenty 
of information on Internet availabil- 
ity and infrastructure as well as the 
dynamics in industries supplying IT 
goods. Eurostat also provides a good 
amount of statistics through the infor- 
mation society statistics survey (ISS). 
ISS is carried out in two main surveys 
pertaining to ‘ICT usage in enterprises’ 
and ICT usage in households and 
individuals’. The aggregate numbers 
can be obtained by breakdowns of age 
group, sex, educational level, employ- 
ment situation and region. However, 
the information provided is limited. In 
terms of e-skills, for example, it is only 
possible to obtain the percentage of 
people who report to have done tasks 
of the type ‘installed a new device’ or 
‘written a computer program' in the 
last three months, in the last year or 
never. Despite the efforts of Eurostat 
in keeping up with the pace of change 
and adapting to new developments 
in ICT, some of the items become 


obsolete relatively fast and have to 
be replaced, which makes it difficult to 
track changes over time. The survey 
is mainly directed to assessment of 
ICT and Internet use in the working- 
age population and thus has limited 
value for education. The ‘ICT usage 
in enterprises’ survey only retrieves 
information on the so-called ‘core 
sectors’ of the economy, which means 
that services such as education are 
not covered by the survey. ISS, there- 
fore, can be used to provide a picture 
of the context in which the effects of 
ICT in education can be assessed but 
would need to be adapted for allowing 
the study of ICT effects in education. 

Studies concerning education at a 
comparative level are carried out by 
the OECD and lEAon a regular basis. 
Their main focus is on the assessment 
of student achievement in different 
competences: reading, mathematics 
and science. These further concern 
themselves with investigation of ICT 
use in education. PISA is probably 
the best known survey of this type. 
It has had important political impact 
and results in PISA are used within 
the OMC to monitor progress towards 
the Lisbon objective (the percentage 
of low-skilled readers is used as one 
of the five benchmarks agreed by the 
Council in 2002). PISA has a specific 
module on ICT. The module has been 
modified in each of the three rounds 
of PISA (2000, 2003, 2006) and will 
probably have a different version in 
2009. It strives to gather information 
from 1 5-years-olds (the PISA target 
group) on the use they make of com- 
puters and their self-reported capac- 
ity for doing certain computer tasks. 
In 2004, the OECD published a report 
specifically looking into PISA and ICT: 
Are students ready for a technology- 
rich world? The report mainly looks 
into the effects of use of ICT in student 
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performance. But it lacks information 
on how the computer has been used 
and in what way because of the limi- 
tations of the ICT module question- 
naire. TIMSS and PIRLS, carried out 
under the auspices of the International 
Association for the Evaluation of 
Education (IEA), also have specific 
information on the use of ICT. In 
TIMSS, for example, information on 
the use of ICT is linked to subject, 
and, therefore, it is more possible to 
explore the impact of the educational 
use of ICT on student performance. 
But we have no information on how 
the computer has been used. 

In terms of thematic studies, there are 
a number of initiatives looking specifi- 
cally into aspects of ICT in education. 
Empirica (2006), in a study financed 
by the European Commission, 
explores the access and use of ICT in 
European schools in 2006. It presents 
information for 25 EU Member States, 
Norway and Iceland, but it does not 
look into student results so it is not 
possible to study this important aspect 
of ICT impact. Another relevant study 
is SITES, which, like TIMSS, is under 
the auspices of the IEA. The survey 
explores the use of computers in 
teaching through sampling teachers, 
principals and ICT responsibility in 
schools. It does not look into student 
achievement, but it does look at the 
perceived impact on ICT in students 
from the teacher's perspective. 

The impact of ICT 
in education 

Balanskat et al. (2006) reviewed sev- 
eral studies on the impact of ICT on 
schools in Europe. They conclude that 
the evidence is scarce and compa- 
rability is limited. Each study uses a 
different methodology and approach, 


and comparison between countries 
has to be done cautiously. Trucano 
(2005) also reviews a series of stud- 
ies on ICT impact in schools. He also 
concludes that the impact of ICT use 
on learning outcomes is unclear and 
calls for the need for more ‘widely 
accepted methodologies and indica- 
tors to assess the impact on educa- 
tion’ (Trucano, 2005, p. 1). In a similar 
line, Cox and Marshall (2007) point 
out that studies and indicators on ICT 
do not reflect sound effects. 

They maintain that this relates mainly 
to three aspects: 

opposing views on ICT and educa- 
tion; 

• different perspectives on/goals 
for innovation in learning/learning 
contexts; 

• missing planning strategies for edu- 
cational change. 

Current approaches for evaluating ICT 
in education are often only focused on 
a few aspects, such as input, utilisa- 
tion and outcome/impact. By the use 
of indicators, they can assess how 
the input (e.g. monetary, infrastruc- 
ture, resources) relates to the impact. 
These models may apply for sev- 
eral purposes, but come too short to 
assess the integration of ICT in poli- 
cies and curricula, particularly because 
they often use a snapshot, one time 
and one level approach. Furthermore, 
evaluation has to care about different 
states in the implementation process 
and analyse changes in the culture 
of the school system — at the micro 
level (pupils) as well as at the meso 
(school) and macro (curriculum/attain- 
ment targets) level. Therefore, a con- 
ceptual framework is needed to look 
into the various dimensions of ICT use 
and to discuss possibilities to measure 
the effects of use of electronic media 
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in education. Such an orientation aims 
at constructing a framework to look at 
the relevant domains and interdepend- 
ence between components related to 
ICT in educational processes from a 
holistic perspective. This paper pro- 
vides a first attempt at an innovative 
approach to the study of the impact 
of ICT/ICT innovation in learning. It 
will further provide a multidimensional 
framework for analysis which can 
locate heterogeneous indicators from 
different studies and data sources. 
This provides a coherent structure to 
guide the exploration of data and the 
map of complex relationships. 

Evaluating different 
stages of implementing ICT 
in education and levels 
of evaluation 

One of the shortcomings of many 
indicator approaches is that they are 
measuring an ‘instance’ within a wider 
historic process, but they are never 
exhaustive, and by being unavoidably 
‘selective’ they can create an incom- 
plete picture of the integration of ICT 
into educational systems. A powerful 
approach to the study of the degree of 
integration of ICT in education makes 
use of such indicators within develop- 
mental models of integration of ICT 
in education. Such models attempt to 
describe potential successive phases 
through which teachers and students 
gradually adopt and use ICT. For 
example, for the context of school 
education, the levels of technol- 
ogy implementation (LoTi) proposed 
by Moersch (1995) identifies seven 
technology implementation levels in 
schools: (a) no use, (b) awareness, (c) 
exploration, (d) infusion, (e) integra- 
tion, (f) expansion and (g) refinement. 
Another model, abstracted from the 
Apple Classroom of Tomorrow (ACOT) 


10-year research project ( 2 ), identifies 
five phases of technology integration 
into schools (see Dwyer et al. , 1991). 
These phases, as described by a more 
recent report on school technology 
and readiness prepared by the CEO 
Forum on Education and Technology 
(CEO, 1999, p. 14); are: entry, adop- 
tion, adaptation, appropriation and 
invention. These models focus on 
what teachers and pupils actually do 
when they use ICT in schools, some- 
thing that the ‘indicators’ approach 
deals with only in a limited way (for 
example, a common utilisation indi- 
cator is the ‘average hours of weekly 
use for teaching’). The above models, 
when used in combination with indica- 
tors such as those described earlier, 
may offer outcomes of more explana- 
tory power regarding the integration 
of ICT in education. They may also 
offer a more solid basis for develop- 
ing models and other instruments to 
study the capacity of educational sys- 
tems to absorb ICT-related pedagogic 
innovations. For example, review- 
ing the above technology integration 
phases in relation to what we defined 
as ICT-related pedagogic innovations 
in schools, one can identify the ACOT 
model's phases of ‘appropriation’ and 
‘invention’ as those offering the most 
promising potential for the diffusion of 
pedagogic ICT-related innovations in 
schools. 

A more recent effort to use indicators 
within a model of ICT integration in 
education was made in the context 
of a project carried out by Unesco’s 
Institute of Information Technology ( 3 ) 
in 2001 . The ‘Morel’s matrix' ( 4 ), which 
was adopted as an instrument for 


( 2 ) See http://www.apple.com/education/k12/ 
leadership/acot/library.html 

( 3 ) See http://www.iite.ru/ 

( 4 ) This matrix was named after Prof. Raymond 
Morel from Switzerland, who developed it. 
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evaluating the degree to which ICT 
has been integrated in an educational 
system, is based on the assumption 
that this process progresses through 
four distinct successive phases: (a) 
emerging, (b) applying, (c) integrat- 
ing and (d) transforming. Unesco has 
further developed this approach to 
help schools determine their stage 
of progress in implementing ICT. 
Variations of the matrix have been 
used in comparative studies of ICT 
implementation at various levels of 
education (see Unesco, 2003a). As 
with the ACOT model, the transition 
from one phase of ICT implementa- 
tion to another in the above matrix 
presupposes the emergence and dif- 
fusion of several types of innovations. 
Pedagogic innovations are implicitly 
assumed to be the driving force in that 
they are sine qua non for any other 
innovation to have a meaningful impact 
on school teaching and learning. 

A somewhat newer version of the 
‘stages' approach is exemplified in 
‘e-maturity’ models (see, for example, 
Butt and Cebulla, 2006; Underwood 
and Dillon, 2004; Underwood et al., 
2007). Such models focus on what 
teachers and pupils actually do when 
they use ICT in schools, something 
that the indicators approach deals 
with only in superficial ways. When 
such models are used to guide evalua- 
tion, in combination with the indicators 
approach, this may offer outcomes of 
more explanatory power regarding the 
integration of ICT in education. They 
may also offer a more solid basis for 
developing models and other instru- 
ments to study the capacity of educa- 
tional systems to absorb ICT-related 
pedagogic innovations. 

Besides the different stages, there are 
several levels to be considered when 
studying the effects of ICT. Indicators 


and emphasis of domains studied may 
vary depending on which of these 
levels are taken into consideration: 
macro, meso and micro levels. The 
macro level refers to aspects at the 
highest level of aggregation. At this 
level, indicators would refer to global 
or national socioeconomic character- 
istics related to the use and integra- 
tion of ICT in education. In a way, 
the macro level could be seen as the 
specific ICT context where meso and 
micro levels are situated. The meso 
level refers to aspects at the institu- 
tional level (school, organisations, 
universities, etc.). The meso level 
refers to aspects related to an inter- 
mediate level that shaped the relation- 
ship between micro and macro level 
aspects. The micro level refers to the 
individual; it portrays individuals in 
their use of ICT. 

These levels present different focuses 
and relate to each other in that lower 
levels are integrated (‘belong’) into 
higher levels (an individual is in a 
school, a school is in a region, a region 
is in a country, etc.). These three levels 
determine the type of indicators that we 
might use within each of the domains. 
Some indicators at the macro level 
or meso level might just be aggrega- 
tions of micro level data. For example, 
the percentage of those reporting the 
use of computers for instruction in a 
country is the result of the aggregation 
of individual (micro level) teachers’ 
answers. If we were to analyse these 
data at the micro level (the impact of 
ICT in a specific individual/teacher, for 
example), the aggregate level indica- 
tor would serve to contextualise his/ 
her answers. Some indicators, on 
the other hand, might be exclusively 
of a specific level, as for example the 
existence of a national policy to have 
all school materials digitalised. 
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Conceptual framework 

All in all, we can say that learning 
practices and teaching for a vari- 
ety of obvious reasons need to be 
assessed in different ways. New tools 
and instruments are required to moni- 
tor both achievements and progress 
made in the context of ICT, but there 
is no clear position yet on adequate 
indicators, instruments and scales for 
measurement. A conceptual frame- 
work would help to alleviate this defi- 
cit. There is a need for a thorough, 
rigorous and multifaceted approach to 
analysing the impact of ICT on educa- 
tion and students’ learning’ (Cox and 
Marshall, 2007; also Kikis and Kolias, 
2005; Aviram and Talmi, 2004). Of 
interest here is that, as early as 1997, 
Collins pointed out that ‘research into 
the contribution of ICT to students' 
thinking and acting reflects the social 
and epistemological beliefs of the 
research community’. This has seri- 
ous implications for evidence-based 
policies. Major policy analyses that 
encompass a wide range of settings 
and look for commonalities and differ- 
ences as a result of systemic condi- 
tions are often missed from most of the 
previous research agendas. Currently 
conducted meta-analyses on ICT 
and attainment suggest that the most 
robust evidence of ICT use in enhanc- 
ing learning was from those studies 
that focused on specific uses of ICT 
(Cox and Marshall, 2007, p. 60). 

Thepurposeofaconceptualframework 
should be to provide an orientation for 
any kind of measurement required in 
the decision-making process. A frame- 
work serves as the basis for modelling 
an appropriate assessment approach 
and the design of methodologies and 
instruments. It connects to all aspects 
of empirical enquiry. When drafting a 
framework, we would therefore expect 


that, contrary to the specific models, a 
conceptual framework acts as a refer- 
ence which is flexible and adaptable 
to the purpose of a study to be car- 
ried out. To take an example: if we 
want to study if technology is having 
a positive impact on educational per- 
formance, a framework would help us 
to identify the various domains in the 
given context to be looked at (such 
as ICT availability and devices used, 
pedagogies applied in which subject 
areas, etc.) and possible perspectives 
to be taken into account (school level, 
individual level, etc.). This is important 
for ensuring that all relevant aspects 
are considered and that a systematic 
approach is followed that is transparent 
and comprehensible for the stakehold- 
ers involved. It provides a holistic view 
and supports the setting of standard 
orientations when defining the evalua- 
tion methodology and selecting appro- 
priate instruments for measurement. 
In more complex evaluation settings, 
when conclusions are to be based on 
a combination of surveys conducted 
by different research teams world- 
wide, it would, ideally, also contrib- 
ute to a coherent common approach 
to the identification of phenomena to 
be analysed and their evidence-based 
interpretation in the light of a common 
understanding of aspects to be stud- 
ied. In the case of the assessment of 
ICT effects in education, this is to the 
benefit of more effective valorisation 
of evaluation studies carried out and 
better quality of analytical work. 

A conceptual framework could further- 
more act as the basis for the design 
of monitoring tools aimed at informing 
policy on the emerging trends, their 
effects and their implications for cur- 
rent or future education. It is therefore 
oriented towards medium- and long- 
term policies and benchmarks defined 
for ensuring effective integration into 
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society. A framework can facilitate the 
construction of models to explain ICT 
effects in education, and for the adap- 


tation of instruments and data sources 
that are further analysed and reported 
(see Figure 1). 


Monitoring ICT impact in education for policy making 



Figure 1: Overall monitoring frame 
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Figure 2: Framework for evaluating ICT in education 



Understanding and evaluating the impact 


A conceptual framework is given in 
Figure 2 for further discussion which 
takes into account the political con- 
text of European education. It covers 
several domains relevant to specific 
EU policy priorities. However, policy 
goals/priorities are presented here as 
an example and could be adapted to 
any other policy priority which might 
be dominant in other countries. The 
framework is divided into domains, 
indicators and stages. 

The domains identified by the concep- 
tual framework here represent the rel- 
evant areas of study. When assessing 
the effects of ICT in education, such 
domains should cover the complete 
range of analytical constructs to be 
studied in the context of the integration 
and use of ICT in education. Ideally, 
each domain should be exclusive 
and not overlap with other domains. 
Based on the literature review carried 
out between 2007 and 2008 relating 
to European projects, case studies 
and research reports, the following six 
dominant blocks were identified in the 
research discussions. 

• Policies: By this term we understand 
any type of strategies relating to the 
implementation of ICT and their 
effective use. This could take place 
at a national policy level as well as 
at an institutional level, such as in 
universities, schools, etc. 

• Resources: This domain refers to 
the ICT infrastructure in terms of 
hardware, software, network capaci- 
ties and any type of digital resources 
used for teaching and learning. 

• Curriculum: By ‘curriculum'we under- 
stand the level of ICT integration in 
the curriculum, including courses on 
how to use ICT effectively. 

• Organisation: This term refers to 
organisational measures to imple- 
ment ICT and its use. One example 


is the use of content/learning man- 
agement systems for educational 
purposes. 

• Teaching practices: This domain 
characterises the use of ICT for 
teaching activities, pedagogical 
practices, etc. 

• Learning: Like the definition pro- 
vided above, ‘Learning’ focuses on 
the use of ICT by the learner (stu- 
dent, etc.). 

It is possible to find specific indicators 
for each of the domains that describe 
the state of the domain and that vary 
from context to context and case to 
case. For example, in the domain 
referring to resources, one possible 
aspect to look at would be ICT avail- 
ability’. As indicated above, the spe- 
cific indicators to look at here would 
be determined partially by the level of 
analysis (macro, meso or micro) to be 
undertaken. As such, at macro level, 
it would be possible to use indicators 
such as ‘broadband penetration’, 'ICT 
availability in the country’ or ‘percent- 
age of educational software sales in 
a country’ among others. At the meso 
level, indicators would be slightly dif- 
ferent and would refer specifically to 
school contexts (or to another meso 
level entity that would be in focus). 
In our example, possible indicators 
would include ‘the presence of LAN in 
schools’ or ‘the percentage of schools 
reporting having educational software’. 
At the micro level, indicators would 
refer to individuals in relation to the 
availability of ICT, for example individ- 
uals reporting on having educational 
software at home and uses made. 

Furthermore, the present framework 
permits the identification of the ICT 
maturity stages. Each of the different 
indicators identified would have certain 
levels that would suggest a specific 
stag e of I CT matu rity . As s uch , co nti n u- 
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ing with our example, ICT resources in 
schools might have reached a certain 
degree that would allow for a ‘trans- 
forming’ stage (let's say all schools in 
a country have an adequate supply of 
ICT tools). However, other indicators, 
for example relating to curriculum, 
might not be as advanced, or have 
no teachers trained in the pedagogi- 
cal use of ICT. These latest indica- 
tors would denote an emerging state. 
Under this scope, the framework pro- 
vides a holistic picture of the range of 
aspects related to ICT. 

It is important to note that the differ- 
ent indicators would have a different 
degree of aggregation depending on 
the analysis that we will want to draw 
from it (see Figure 1). The framework 
provides the pre-stage for the analysis, 
allowing stakeholders to see the rele- 
vant aspects in a holistic picture before 
a specific analysis is carried out. As 
such, individual reporting ofthe number 
of computers at home, for example, 
can be aggregated at the national level 
to analyse country-specific patterns in 
relation to use and possession, or can 
be used at the individual level to carry 
out studies on the use and possession 
of ICT by individuals in relation, for 
example, to their age. Our framework 
permits the review results ofthe analy- 
sis in light of the ‘greater scenery of 
ICT' within a given setting. This facili- 
tates the consideration of aspects not 
specifically accounted for in the origi- 
nal level of analysis, but which might 
play an important role in understand- 
ing the results. 

Outlook 

Conceptual frameworks are impor- 
tant tools for orienting and evaluating 
policy decisions. They offer policymak- 
ers dimensions for consideration when 
evaluating the effectiveness of policy 


interventions and provide a basis for 
further decisions. The framework pre- 
sented in this paper builds a compre- 
hensive model for the analysis of ICT 
effects into the educational process 
from various levels and perspectives. 
It establishes a structure for reflecting 
on relevant indicators. The framework 
takes into account different levels of 
analysis allowing therefore for dif- 
ferentiation in scope. The framework 
further introduces different stages of 
implementation. This allows policy- 
makers to acquire a holistic view on 
policy changes and the effects these 
have on different actors within the 
educational system. A holistic view is 
an essential aspect for policy evalua- 
tion because it can disclose the matu- 
rity ofthe implementations of policies. 

In brief, the paper proposes that in 
order to deepen our analysis of the 
impact of ICT on education, we need 
to shift our attention from technology 
per se to processes and skills teach- 
ers and learners are currently apply- 
ing. This will allow us to identify and 
explore conditions and factors that 
are shaping the way ICT is used in 
education. Under this perspective, 
we need to shift from approaches 
that exclusively monitor macro level 
aspects to an integrated model where 
the three different levels are consid- 
ered in conjunction. Such a compre- 
hensive approach to the study of ICT 
effects and their impact on education 
needs to be considered in a coher- 
ent manner. The proposed framework 
allows for the integration of differ- 
ent levels and types of data sources. 
It is important to bear in mind that 
there appears to be a need to reflect 
beyond pure observations and evalu- 
ate more concretely institutional con- 
texts of learning (schools, university, 
etc.), learning situations and teaching 
processes to determine under which 
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circumstances ICT-based activities 
can enhance learning and improve 
skills. Due to the complexity involved 
in mapping factors/variables on to one 
another, the evaluation of the causes 
of the observed impacts requires a 


degree of qualitative interpretation. 
It is highly recommended that the 
actors engaged in the process define 
the scope for evaluation and on such 
bases interpret the results. 
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ICT to improve quality in education — 

A conceptual framework and indicators 
in the use of information communication 
technology for education (ICT4E) 

Marcelo Cabrol and Eugenio Severin O 

Inter-American Development Bank ( 2 ) 


1. Introduction 

The use of information and commu- 
nication technologies (ICT) in educa- 
tion is no longer optional. Asubstantial 
change in society and individuals ( 3 ) 
has occurred thanks to development 


(') We express our thanks for the revisions 
and corrections by Carla Jimenez (IADB). as 
well as comments and suggestions by Juan 
Enrique Hinostroza, Claudia Peirano, Maria 
Paz Dominguez, Francesc Pedro, Friedrich 
Scheuermann, Seong Geun Bae and Michael 
Trucano. 

( 2 ) The Inter-American Development Bank 
Technical Notes encompass a wide range of 
best practices, project evaluations, lessons 
learned, case studies, methodological notes 
and other documents of a technical nature that 
are not official documents of the bank. The 
information and opinions presented in these 
publications are entirely those of the author(s), 
and no endorsement by the Inter-American 
Development Bank, its Board of Executive 
Directors or the countries they represent is 
expressed or implied. 

( 3 ) ‘In this technological environment, computers 
have become an integral part of our societies 
and our lives, transforming such diverse matters 
as the way we work and relax, how businesses 
■operate, the conduct of scientific research, 
und the ways governments govern. They are 
integrating into other technologies — in cars, 
phones and many other things that used to be 
low-tech'. There is'every reason to suppose that 
the pace of technological change will continue 
though we cannot say precisely in which forms 
and directions.’ (OECD, 2008) 


in ICT, its penetration into the struc- 
tures of production, knowledge man- 
agement, communication and culture, 
the demand for new skills and compe- 
tencies and the loss of importance in 
others. In addition, there has been a 
change in ways of approaching and 
understanding the world and devel- 
opment of new industries. For all 
these reasons, schools, countries and 
regions are compelled to develop new 
initiatives that incorporate ICT tools in 
teaching and learning, so that educa- 
tion systems can succeed in linking 
the new demands of the knowledge 
society with the new characteristics of 
learners ( 4 ). 

Some education systems in Latin 
America have overcome the chal- 
lenge of access to education and are 
now confronting the demand for qual- 
ity improvement; some systems face 
significant challenges in attempting to 
include all children in the learning proc- 
ess; others require more radical solu- 


( 4 ) ‘Economic theory describes three factors 
that can lead to increased productivity: capital 
deepening (that is, the use of equipment that is 
more productive than earlier versions), higher- 
quality labor (a more knowledgeable workforce 
that is more productive), and technological 
innovation (the creation, distribution and use of 
new knowledge).’ (Kozma, 2008) 
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tions to support student learning while 
designing strategies within already 
fragile institutions. In any case, prop- 
erly implemented ICT projects offer 
an alternative for implementation and 
impact on student learning ( 5 ), espe- 
cially with new millennium learners 
(NML) ( 6 ). 

Nevertheless, most current evalua- 
tions have no conclusive information 
to inform decision-makers on how 
ICT can improve the quality of edu- 
cation ( 7 ). The lack of clarity about 
different options and impact areas of 
ICT use in education is an obstacle 
for the development of successful 
projects. 


( 5 ) 'All the studies reviewed have identified 
a range of important wider benefits of ICT on 
learning. These include the positive impact of 
ICT on student motivation and skills, independent 
learning and teamwork. Increased motivation 
leads to more attention during lessons which can 
be exploited by the teacher. Aspects for more 
individualised learning were described in a variety 
of ways. Students learn more independently, at 
their own pace and according to their needs. 
They also take more responsibility for their 
own learning process. As seen, ICT can benefit 
likewise academically strong and weak students 
and students with special needs.’ (Balanskat et 
al„ 2006) 

( 6 ) 'In times when a lot of emphasis is put on 
the effectiveness of teaching, more attention 
should be devoted to the changes occurring in 
pupils as they increasingly become NML. Their 
emergence claims for a reconsideration of ICT- 
based educational innovations putting pupils’ 
new attitudes and expectations, as well as 
transformed competences at the very centre.' 
(Pedro, 2006) 

( 7 ) The exercise to establish a ‘knowledge map' 
developed by the World Bank infoDev (Michael 
Trucano, 2005) showed how, beyond the large 
investments made in many countries to use 
ICT in education systems, data to support the 
affirmation of its role in improving education are 
limited and debatable. 


Inadequate assessment of the incor- 
poration of ICT initiatives in education 
is in many cases a result of intuitive 
and unsound development but also 
relates to the lack of specific tools 
that would give confidence to meas- 
ure these impacts adequately sepa- 
rate from a myriad of other variables 
present in educational processes, 
which are dynamically affected by the 
introduction of ICT. 

It is very likely that this lack of instru- 
ments is a natural consequence of the 
emerging development in this process. 
Considering that the personal com- 
puter has been in existence for only 
30 years, and that the first computers 
that came to some schools did so only 
about 20 years ago, it is only logical 
that we still have many unanswered 
questions about how ICT can achieve 
the best contribution towards improv- 
ing the quality of education. 

In fact, recent literature has drawn 
attention to the innovation phenom- 
enon in educational practices incor- 
porating ICT, with the caveat that so 
far the greatest amount of experience 
has been limited to ‘computerisation’ 
of processes and practices, which 
continues to repeat the same actions 
of the past, but now with the support 
of computers and other technological 
devices. The predictable consequence 
is that impact on results will be quite 
limited ( 8 ). 

The use of ICT in the context of dis- 
ruptive innovation and comprehen- 
sive intervention regarding the above 


( 8 ) ‘Many of today’s schools are not teaching the 
deep knowledge that underlies innovative activity. 
But it is not just a matter of asking teachers to 
teach different curriculum, because the structural 
configurations of the standard model make it very 
hard to create learning environments that result 
in deeper understanding.’ (Sawyer, 2008). 
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practices is probably a better progno- 
sis for changing results. Nevertheless, 
even less data, assessments or stud- 
ies are available on this issue ( 9 ). 

This document presents a general 
conceptual framework to support the 
design, implementation, monitor- 
ing and evaluation of projects where 
information and communication tech- 
nologies have been incorporated to 
improve education quality. 

One of the main challenges in the use 
of ICTin education is the lackofindica- 
tors that offer clear criteria and objec- 
tive information to allow policymak- 
ers to make the proper decisions ( 10 ). 
Projects have not always considered 
rigorous evaluation processes and 
in those instances where they have, 
ICT impact on learning has not been 


( 9 ) ‘Schools should use computers and related 
technologies to help students who are poorly 
served, or not served at all, by the current 
technology of education — that is, by the schools 
most of us grew up with. In addition, elementary 
and secondary students ought to use computers, 
the Internet and other digital tools directly, 
not necessarily through a school. In these 
ways, schools, students and families will help 
promising computer-based technologies grow 
and improve. The schools can pay a huge price 
for not changing in time to accommodate new 
technologies.’ (Christensen et at, 2008) 

('“) The World Summit on the Information Society 
(WSIS) concluded that: ‘We must develop a 
realistic plan for evaluating results and setting 
benchmarks (both qualitative and quantitative) 
at the international level, through comparable 
statistical indicators and research findings to 
monitor the implementation of the action plan 
goals and objectives, taking into account national 
circumstances.’ (WSIS 2005) 


the focus ("j. Lastly, the project offer- 
ing is so vast that there is no common 
framework that can be both flex- 
ible and broad enough to include the 
diverse nature, contexts and different 
stages of projects. 

The main hypothesis of the framework 
is that the goal of all education projects 
is to improve student learning, regard- 
less of whether they are children or 
adults. The goal expected and meas- 
ured in these projects should then be 
impact(s) on learning and changes 
brought about by implementation and 
enabling such learning. 

Learning outcomes can be broadened 
by putting children at the centre of the 
learning process. It is necessary to 
consider improvements in students’ 
involvement in and commitment to 
learning as the initial result. This plays 
a direct role in curricular learning 


( u ) See, for example, the conclusion of the 
World Bank evaluation of an ICT programme 
in Colombia: The main reason for these (poor) 
results seems to be the failure to incorporate 
the computers into the educational process. 
Although the programme increased the number 
of computers in the treatment schools and 
provided training to the teachers on how to use 
the computers in their classrooms, surveys of 
both teachers and students suggest that teachers 
did not incorporate the computers into their 
curriculum’ ( The use and misuse of computers 
in education evidence from a randomized 
experiment in Colombia, Barrera-Osorio and 
Linden, 2009). Also in the Enciclomedia Project 
in Mexico: ‘no significant differences were found 
in the knowledge skills, implementation and 
evaluation of content among children who used 
Enciclomedia and those who did not have such 
equipment. Even children from 6th grade who 
did not use that technology had a better result by 
reaching 1 .48 over 1 .23 points over those who 
did have such a tool, Whereas in the application 
of content learned, those first gained 2.15 points 
to 2.11 for those who did have this tool. Those 
5th grade students without Enciclomedia were 
best evaluated with 1.83 points on 2 of their 
classmates with this equipment’, Libro Blanco 
Enciclomedia, ILCE, 2007. 
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improvement and can be observed 
in the participation and continuance 
of students in the learning process, 
and with improvement in teaching 
practices and learning processes 
as well. It also takes into account 
that these changes in practices and 
improvements are directly linked to the 
impact and the development of either 
general skills or '21st century skills’, 
including an understanding of ICT skill 
acquisition ( 12 ). 

The monitoring and evaluation proc- 
ess should be considered more care- 
fully and rigorously as a substantial 
component of each project, much 
more than it has been thus far, to 
account for such impacts. Monitoring 
and evaluation processes must be 
incorporated as an integral part of the 
process itself. Review of key informa- 
tion before (baseline), during the proc- 
ess (monitoring) and at the end of the 
project (final evaluation) is fundamen- 
tal to the proposed framework. The 
use of indicators to measure the sys- 
tem’s level of development and matu- 
ration will be an indispensable tool 
for making policy decisions based on 
solid data and targeted knowledge. 

The proposed framework identifies 
five domains (inputs) that should be 
considered in an education system or 
in each specific project, its planning 
processes and products, and those 
processes that, though not directly 


( ,2 ) In the case of students from low-income 
families, the flexibility of schools is even smaller. 
Wealthier schools attract the best teachers, 
leaving the least prepared teachers to schools 
in poor and remote areas. [...] Consequently, 
these systems perpetuate social inequalities, 
lose excellent students as victims of boredom, 
increase the cost of education through the high 
dropout and repetition rates, and pass the cost 
of training their graduates to employers or other 
systems. (Haddad and Drexler, 2002) 


involved, can be affected by the devel- 
opment of the project ( 13 ). 

Application of this framework and 
indicators at different levels of edu- 
cation systems (national or subna- 
tional) aims to provide a holistic and 
integrated vision of ICT incorporation 
in order to support decision-making 
regarding actions that can or should 
be made based on the available 
information, taking into considera- 
tion all necessary areas or domains 
(inputs). 

At the specific project level, use of 
diverse quantitative and qualitative 
methodologies for data collection and 
observation will provide a set of indica- 
tors. This evaluation will allow meas- 
urement of the project’s efficiency 
and monitoring of its development 
by those carrying out the project and 
other stakeholders, making it easier to 
determine best practices and promote 
the development of new initiatives for 
use of ICT in education areas. 

This framework has been developed 
taking into account empirical informa- 
tion available from past Inter-American 
Development Bank experience and 
from other experts in ICT education 
project implementation. 

Considering that every ‘ICT in educa- 
tion’ project implements different lines 
of action, the framework is broad in 
nature, allowing different variables to 
be reviewed and selected (like a road- 


( 13 ) Since computer availability alone will not 
have an impact, policymakers and project leaders 
should think in terms of combinations of input 
factors that can work together to influence learning. 
Coordinating the introduction of computers with 
national policies and programmes related to 
changes in curriculum, pedagogy, assessment 
and teacher training is more likely to result in 
widespread use and impact. (Kozma, 2005) 


86 


Improving quality in education 


map) depending on direct or indirect 
involvement in the project and how 
they could be affected by it. 

Regardless of the variables and com- 
ponents included in the project, the 
goal (and objectives) should be linked 
to the improvement of learning and 
its implementation should take into 
account monitoring and evaluation 
mechanisms linked to the objectives. 
A good evaluation will allow results 


from one ICT education project to 
be compared to other projects (ICT- 
related or not) in order to evaluate the 
efficiency of the investment. 

This document should be considered 
a working paper within the conceptual 
framework, which will be improved 
through the development of new 
projects and continually updated due to 
the constantly changing nature of ICT 
education processes and products. 
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As shown in the table, the framework 
includes the following elements. 

Student learning, as the main 
goal of all project implementation. 
Students must be considered the 
direct beneficiaries of any ICT4E 
initiative, regardless of whether they 
are children or adults. 

• The Inputs refer not only to project 
lines of action but also to fac- 
tors that could be affected by its 
implementation. 

• The Processes and products 

are those elements that will be 
modified by the project and should 
demonstrate the results of the 
implementation. 

The projects' Impact and the con- 
ditions that allow such outcomes 
are measured broadly with different 
variables. 

Development stages: four stages 
are described which will impact the 
design, implementation and evalua- 
tion of the projects. 

• The process of Monitoring and 
evaluation includes different 
sources of data and information. 

The elements included in the frame- 
work are described below. 

1. Student learning 

Student learning is the purpose and 
main goal of an education system’s 
actions and must remain so regarding 
use of ICT in educational processes. 

In each specific project, students are 
direct beneficiaries, so the expected 
results should be directly linked to the 
learning that the project explicitly aims 
to impact or which will be indirectly 
impacted by the project’s action. 

The project’s impact (positive, nega- 
tive or no change) and its effective- 
ness will depend on evidence of 


change that can be demonstrated in 
students learning. 

2. Impact 

Results 

1. Practices 

The use of ICT in education implies 
the reasonable expectation that modi- 
fications in teaching methodologies 
and student learning processes will 
occur ( 14 ). 

ICT offers a unique opportunity for 
access and knowledge construction. 
In order to achieve effective, compre- 
hensive use of ICT in education devel- 
opment of new learning practices, 
strategies and methodologies must 
be put into place ( 15 ). A review of the 
literature indicates that, in instances 
where ICT has been incorporated 
as an additional tool to 'maintain the 
status quo,’ educational impacts are 
scant or non-existent. 

This is an important field for innovation, 
where ICT4E plays an important 
catalysing role. The link between 


( 14 ) ‘When learning scientists first went into 
classrooms (Sawyer, 2006), they discovered 
that most schools were not teaching the deep 
knowledge that underlies knowledge work. By 
the 1980s, cognitive scientists had discovered 
that children retain material better, and are able 
to generalise it to a broader range of contexts, 
when they learn deep knowledge rather than 
surface knowledge, and when they learn how to 
use it in real-world social and practical settings. 
Thus, learning scientists began to argue that 
standard model schools were not aligned with the 
knowledge economy.’ (Benavides et al., 2008) 

( 15 ) ‘Measuring changes in learning and 
teaching processes is a time-consuming task, 
but one which may yield valuable results. 
Knowing how educational technology changes 
teaching practices, as well as the ways in which 
students learn, is fundamental for evaluating its 
effectiveness and for developing better tools.’ 
(Balanskat et al., 2006) 
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teaching and learning practices 
and the growing daily interaction of 
students with digital, multimedia and 
interactive environments make this 
a key aspect of the framework and 
an important element to connecting 
projects with expected results ( 16 ). 

2. Student involvement 

One of the fundamental components 
of educational processes is student 
commitment. Although it may be obvi- 
ous, the motivation and ongoing par- 
ticipation of students are necessary 
for project success. Furthermore, 
student motivation and enthusiasm in 
activities have a positive impact, not 
only with respect to potential learn- 
ing results and development of new 
competencies but also to the learning 
environment, in stakeholder expecta- 
tions and results for student promotion 
from one level to another. These proc- 
esses also generate change regard- 
ing the motivation and expectations of 
parents and teachers. Both are inter- 
twined with student motivation and 
expectations, resulting in the ongoing 
development of learning, 

Data on attendance, repetition, pro- 
motion and drop-out rates are usually 
available and facilitate the performance 
of straightforward impact analysis. 
Measuring motivation requires other 
instruments which, when applied cor- 
rectly, can yield important information 
about the effects of ICT4E projects. 


( 16 ) ‘One of the fundamental lessons to be learnt 
from European, North American and Australian 
experiences over the last 20 years has been 
that those responsible for helping people learn 
must be confident in the appropriate use of new 
technologies if the process is to be successful. 
Introducing new technologies into places of 
learning should involve a fundamental shift 
whereby the role of teachers becomes less 
didactic and more that of facilitating individual 
learning processes.’ (Unwin, 2005) 


Impact 

3. Student achievement 

A country’s education curriculum 
determines the knowledge and skills 
that students should achieve for each 
grade as well as tasks required of 
teachers and schools. The first area 
where impact is evident in ICT4E 
projects is in learning associated with 
a specific school subject or topic, or 
how the curriculum content is divided 
according to learning aims or expected 
competencies for each student. 

Typically, this impact has been evalu- 
ated in subjects such as language, 
mathematics and science, since these 
are the subjects evaluated in most 
standardised tests (focus groups or by 
census) and, therefore, data are avail- 
able in many countries (e.g. standard- 
ised tests such as TIMMS and PISA). 
Even though these instruments have 
had a small, limited field of meas- 
urement to date (limited to only cer- 
tain skills and content), studies have 
revealed positive but moderate corre- 
lations between ICT projects and test 
results. 

There are some challenges in coun- 
tries that do not have national tests or 
participate in international standard- 
ised tests. In these cases the project 
could develop ad-hoc standardised 
tests to be administered before, during 
and after the project implementation 
(baseline and evaluation) or among 
groups that do or do not participate in 
the project (control and comparison 
groups). 

A lack of rigorous studies in this 
area has made it harder to prove 
the reasonable expectation that a 
country’s investment in ICT projects 
can improve learning in different 
subjects. Therefore it remains to be 
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seen whether this impact is significant, 
and, if so, on what subjects. More 
important yet is the lack of clarity as 
to what impacts can be reasonably 
expected in projects according to their 
stage of development or maturity. 

This task is especially complex 
because the introduction of ICT into 
education processes is often accom- 
panied by modifications in teaching 
methodologies. In fact, this is what is 
intended; with the introduction of ICT, 
old methodologies could have little or 
no impact. 

Evidently both people and govern- 
ments reasonably expect that use of 
ICT in education (usually a complex 
and expensive process) will improve 
student learning, and this needs to be 
proven empirically. 

4. Skills and competences 

It is fairly common to point out that ICT 
use in education has an impact on the 
development of new skills and com- 
petencies in students. These compe- 
tencies have often been described as 
‘21 st century skills’ due to their impor- 
tance in a knowledge society age ( 17 ). 

There is extensive literature describing 
these competencies and it is therefore 
easy to consolidate a group of gen- 
eral competencies required by stu- 
dents that will eventually develop fully 
with the use of ICT. They have been 
grouped into three major areas: critical 


0 7 ) ‘To participate in this global economy and 
to improve their standard of living, students will 
need to leave school with a deeper understanding 
of school subjects, particularly science, 
mathematics and technology. They will need 
skills necessary to respond to an unbounded but 
uncertain 21st century to apply their knowledge 
to real-world situations, to think critically, to 
collaborate, to communicate, to solve problems, 
to create and continue to learn.’ (Kozma, 2008) 


thinking and problem solving; creativ- 
ity and innovation; and communication 
and collaboration. Development of ICT 
competences is also considered. 

Until now, evaluation has not been 
particularly exact and has been mostly 
conducted through qualitative studies, 
interviews and perception surveys that 
collect information/data on the vision 
of students, or through structured 
observation exercises. Nevertheless, 
more objective tools will be developed 
over time that will allow for more rigor- 
ous evaluation exercises. 

One of the components of the OECD 
new millennium learners project is 
developing ICT competencies for a 
working definition framework and 
tools for evaluation. Another initiative 
working towards similar objectives is 
the alliance supported by CISCO, Intel 
and Microsoft and a group of univer- 
sities and international institutions: 
'Transforming education: assessing 
and teaching 21st century skills’. 

Information and communications 
technologies are instruments that are 
a regular part of a range of work and 
development opportunities. Even a 
basic understanding of ICT use can 
result in opportunities for access and 
growth, both personally and profes- 
sionally, which can make the difference 
in a country's overall development. 

ICT skills and competencies are a 
clear objective in any project involving 
the use of ICT in education; therefore it 
is necessary to evaluate the effective- 
ness of each project. To perform these 
tasks, standardised tests will be used 
alongside IDB’s own validated test 
to evaluate student ICT skills before, 
during and after implementation of 
activities in primary education. 
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3. Development stages 

Clearly, the type of projects to 
develop and evaluate (as well as the 
impacts expected) will depend on the 
respective stage of development in 
the use of ICT in and the educational 
context where each project will be 
applied ( 18 ). 

The development stage reached 
through incorporation of ICT into 
education systems is strongly cor- 
related with the type and depth of 
potential changes in application con- 
texts. Thus, the intensity of use and 
the impact increase to the extent that 
efforts toward incorporation are sus- 
tained over time. 

Following Morel’s Matrix (2001), four 
project phases are proposed which 
are vital in the project’s design, imple- 
mentation, follow-up and evaluation 
steps, and in the follow-up of compara- 
ble education systems. Therefore, by 
analysing the indicators described in 
the ‘Processes and products' column, 
you can determine the development 


( ,s ) ‘Countries which are presented in the 
initial stages of ICT incorporation in education 
have different assessment needs than those 
who already have a long tradition of use. For 
example, initially it is important that teachers 
and students have access to software and 
hardware and that they have acquired basic 
skills in computer science. Countries which 
are at the initial stages of ICT incorporation 
in education have different assessment needs 
than those who already have a long tradition 
of ICT use. For example, initially it is important 
that teachers and students have access to 
software and hardware and that they have 
acquired basic skills in computer science. In 
the case of countries at more advanced stages, 
other considerations such as management 
of educational innovations, changes in 
educational curricula and other organizational 
changes in schools, and ongoing support 
and training for staff are more important". 
( Manual for the production of statistics on the 
information economy, UNCTAD, 2008) 


stage ofthe project (emerging, apply- 
ing, integrating and transforming) and 
inform the expected outcome with 
results indicators. 

For example, you can generically 
describe these steps for each domain 
considered in the general framework, 
in the table. 

The table operates in practice as 
a section for reading the indicators 
present in a system or project, which 
allows for ascertaining maturity or 
stage of development. 

Once this section has been applied 
to each system or project, reading 
it may provide criteria for decision- 
making regarding the domains regis- 
tering greater or less progress and, 
therefore, the kind of priorities that 
could lead the development of new 
actions. 

Definition of development stages is 
directly related to reasonable expec- 
tations for the impact that ICT has on 
educational systems, particularly with 
respect to learning, skills and student 
competences. It is therefore possible 
to enter into the table below some 
examples of the kind of results that 
can be found in education systems 
or in project target groups. Analysis 
of indicators will therefore depend on 
each stage of development. 

Until now, limited and partial invest- 
ments in ICT (implying very small 
changes in inputs) were rarely 
expected to involve changes that 
can quickly translate into new and 
improved skills and competences in 
students. Applying this framework 
has allowed us to recognise that the 
achievement of significant impacts is 
the result of a development process 
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that requires a broad vision, compre- 
hensive, integrated implementation 
and development time in order to 
exhibit genuine impact. 

4. Domains or inputs 

Domains or inputs considered in 
project design and evaluation include 
the following: 

1. Infrastructure 

a. Physical: Initiatives associated 

with provision of infrastructure nec- 
essary for the use of and access to 
ICT, e.g. laboratories, libraries and 
furniture. 

b. Equipment: Equipment planned 
for the project or considered part of 
the project (even if not conceived as 
a direct part of the project) includes 
computers, printers, projectors and the 
conditions included in the purchase 
and use of those items, e.g. guarantee 
and service support. 

c. Connectivity: Access to Internet 
and networks that allow their use 
for education purposes; bandwidth 
access, connection stability and tech- 
nologies that facilitate better online 
traffic and provide privacy protection 
filters for content accessed by stu- 
dents. Implementation of a reliable 
local network structure that is safe and 
accessible. 

d. Support: Activities aimed at admin- 
istration, maintenance and repair of 
equipment as well as problem-solving 
related to project activities and techni- 
cal support for users. 

2. Contents 

a. ICT curriculum: Initiatives linked 
to the implementation and/or adap- 


tation of curriculum content in ICT or 
other subjects (in the use of ICT). 

b. Content: Digital or analog material 
aimed at teaching and learning with 
technology tools, e.g. encyclopedias, 
manuals, textbooks, books, guides, 
videos and hypertext. 

c. Tools: Software development or 
support initiatives for development of 
teaching and learning processes; e.g. 
productivity applications, virtual simu- 
lators and modeling. 

d. Information systems: Aimed at 
supporting implementation and distri- 
bution of management and education 
information systems at the school, 
country and regional levels, as well as 
those that allow monitoring of educa- 
tional projects and their stakeholders, 
including curriculum, pedagogies and 
possible models of use ( 19 ). 

3. Human resources 

a. Teacher training: Initial and in- 
service training associated with the 
adoption, adaptation and updating of 
curriculum and practices for the inte- 
gration of ICT into education. 

b. ICT competences: Training activi- 
ties for the acquisition and/or certifi- 
cation of specific ICT skills, general 
education, and productivity and com- 
munication tools. 


( ,9 ) 'Clearly, compared to the traditional structure 
of the Internet, with few transmitters and many 
receivers, a new platform begins to be adopted 
where web applications are easy to use and 
allow for many transmitters, many receivers and 
a significantly higher information exchange rate. 
Some of the most common resources are having 
an impact in teaching models based on online 
technologies such as blogs, wikis and others.’ 
(Cobo Romany and Kuklinski, 2007) 
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c. Use of ICT for education: 

Training initiatives for the specific 
use of ICT in educational 
contexts ( 20 ). 

d. Pedagogical support: Efforts to 
provide educational support and follow 
up for participants, guidance or tutor- 
ing service developed for implementa- 
tion of proposed activities. 

4. Management 

a. Administration: Structures and 
strategies for system and project 
management and administration for all 
levels considered (school, province, 
country and region) as well as the rela- 
tionship with other institutional stake- 
holders associated with the project 

e. g. strategic allies and donors. 

b. Information dissemination: 

Activities aimed at providing informa- 
tion about project results, strategies 
and actions and involving all potential 
interested stakeholders and benefici- 
aries of the project. 

c. Community involvement: How 

scope, strategies and actions are 
communicated. How all actors con- 
cerned and potentially affected by the 
project’s development are involved. 
Actions that promote (and allow for) 
the active participation of community 
members and families in the develop- 
ment (and as direct beneficiaries) of 
the project. 

5. Policy 

a. Planning: The project’s prior- 
ity (short or long term) in the context 
of other initiatives, plans, projects or 
actions, including visibility (under- 


( 20 ) Particularly important here is Unesco’s work 
in the development of the use of ICT in education 
and its standards for teachers. 


stood as the ownership level with the 
success and objectives of those lead- 
ing the project). 

b. Budget: Long-term budget needed 
for operational continuity and devel- 
opment of complementary initiatives 
required for the project’s success. 

c. Legal framework: Actions to adjust 
and adapt the rules and regulations to 
enhance and improve the impact of 
the initiative and minimise the risks. 
Includes measures to improve the 
safety and security of minors, regula- 
tions associated with industries and 
copyright protection 

d. Incentives: Plans and programmes 
designed to (positively or negatively) 
underscore beneficiary commitment 
and the results of the project expected 
by its participants. 

5. Processes and products 

Processes and products being pro- 
posed to allow the framework to sup- 
port the design, implementation and 
monitoring of specific projects devel- 
oped to incorporate the use of ICTs for 
educational purposes. 

For example, listed below are some of 
the products and processes that may 
typically be considered as part of these 
projects and whose observation and 
monitoring will reveal how each contrib- 
utes to achieving the expected results. 

1. Infrastructure 

a. Amenities: Specific references 

about the technical characteristics 
of the equipment. The relationship 
between product characteristics and 
specific reasons why the equipment 
was selected; distribution and the final 
characteristics of the equipment as it 
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is implemented. Also included here 
is the connection with other existing 
equipment indirectly related to the 
success of the project. Characteristics 
and conditions of connectivity. 

b. Implementation process: Des- 
cription of project logistics, location and 
equipment distribution. Additionally, 
specifics on the procedure for equip- 
ment selection, purchasing, distribu- 
tion and integration/implementation in 
projects. Also included are references 
to the investment made in the context 
of the project essential to its success, 
such as classrooms or buildings (even 
when they have not been a project- 
specific component), as well as calen- 
dars and systems in use by ICT users 
and their availability. 

c. Helpdesk: Describes systems 

installed to lend support to indirect 
and direct project beneficiaries in 
the event of technical and pedagogi- 
cal difficulties. It will provide the user 
rate, response time, mechanism used, 
most common difficulties, the best- 
rated responses and other indica- 
tors describing support available to 
participants. 

2. Resources 

a. Curriculum development: Work 
developed to connect curriculum to the 
learning goals and project objectives 
associated with ICT4E. Inclusion of ICT 
in the curriculum at the different levels 
as a competency or as cross-cutting or 
vertical content, learning goals specifi- 
cally proposed by the stakeholders. 

b. Learning organisation: Des- 

cription of how learning activities are 
structured and organised, including 
how the curriculum is developed (inte- 
grated or separated from other the- 
matic areas), how often and at what 
time of day ICT is integrated into the 


curriculum, pedagogical approach(es) 
at the institutional level as well as 
knowledge management strategies. 

c. Availability of resources: Levels 
of access to educational resources 
from direct and indirect beneficiaries; 
whenever possible underscore rele- 
vance and importance with respect to 
project objectives. 

d. Access and use: The opportu- 
nity for and simplicity of access to the 
information and management systems 
by the beneficiaries (direct or indirect), 
whenever possible, provide their rel- 
evance to and the quality of the pro- 
posed objectives. 

3. Human resources 

a. Teacher performance: Describes 
teacher background information perti- 
nent to the project: e.g., performance, 
planning activities, student:teacher 
ratio, performance evaluation and 
incentives. 

b. ICT experience: Previous experi- 
ence with ICT in educational use, both 
in and outside the classroom. 

c. Models for educational use: 

Characteristics of ICT training to stake- 
holders in order to capitalise on the 
use of ICT in educational contexts. 

d. Education support system: 

Mechanisms aimed at motivating and 
lending support to the work of different 
stakeholders involved in the project, 
such as tutoring or assistantships for 
teachers, personal or online support 
plans, training resources, mutual com- 
munication among peers and guides 
for families. 
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4. Management 

a. School organisation: The way the 

project is integrated into the overall 
institutional scheme of the school, how 
many hours each teacher spends on it 
and systems aimed at organising and 
supervising the project’s operation. 

b. Management systems: Insti- 

tutional framework, systems and 
mechanisms implemented by the 
project, orthatthe project modifies and 
impacts and which allow for follow-up 
of project activities and objectives. 

c. Systems use: Opportunity for and 
simplicity of access to the information 
and management systems by the ben- 
eficiaries (direct or indirect), whenever 
possible stating relevance to and qual- 
ity of the proposed objectives. 

d. Community attitudes and expec- 
tations: Actions involved in the 
project’s implementation aiming to 
include the initiative in its develop- 
ment context, introduction of partici- 
pants (direct or indirect) to the project, 
communication with those involved in 
the project who facilitate the project’s 
implementation. Also describe how 
the project considers the impact on 
the community, particularly regarding 
students’ families. 

5. Sustainability 

a. National (subnational) plans: 

Displays the existence or lack of 
national plans that comprehensively 
maintain and describe the use of ICT 
in education systems, linking them to 
each other and to the rest of the goals 
and policies, and to the development 
strategies as well. 

b. Budget: Different budget sources 
and procedures that are directly or 
indirectly involved in the project's oper- 


ations. Any difficulties with the proce- 
dure and future financing plans should 
be described. The expenses entailed 
by the project should be noted, speci- 
fying one-time purchases as well as 
recurring purchases that will therefore 
be part of the project in the future. 
Mechanisms recommended to secure 
funding in the future. For long-term 
implementation, the project's strengths 
and weaknesses and how the project 
itself plans to address them. This will 
include the total ownership cost as 
proposed by GESCI ( 21 ). 

c. Priority and visibility : The position 
of those responsible for the project as 
well as project objectives and the pro- 
motion of such activities. 

d. Legal framework: Description of 
regulations associated with project 
implementation. 

e. Incentives plans: Programme or 
incentive plans associated with the 
project’s beneficiaries and objectives. 

6. Evaluation 

The Conceptual Framework is not 
proposed as an evaluation model, nor 
does it develop specific assessment 
instruments. It should work as a guide 
to consider the elements involved in 
ICT for education projects. The evalu- 
ators using the Conceptual Framework 
should then apply and develop the 
adequate evaluation models and instru- 
ments depending on the context. 

1. Baseline 

The data that inform the processes and 
products before the project’s imple- 
mentation and by which the project 
impact can be measured. 


( 21 ) Global e-school and communities initiative 
http://www.gesci.org/ 
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The baseline is concerned with data 
that allow for identification of indicator 
status at the system level upon start- 
ing the application or before project 
implementation. From these initial 
data (sum zero), system progress or 
project action impact will be meas- 
ured, once they are implemented. 

The baseline should take into con- 
sideration the systems level, a broad 
set of indicators that facilitates pre- 
cise analysis of ICT incorporation 
status. At the project level, you should 
select those indicators that explicitly 
impact the project’s objectives, includ- 
ing those linked to student learning. 
Wherever possible, however, the data 
for all processes should be taken into 
consideration to facilitate documenta- 
tion of unforeseen impacts. 

2. Follow-up and monitoring 

When applied at the system level, 
steady action is required that may be 
implemented to ascertain changes 
occurring due to various actions aimed 
at incorporating ICT into education 
systems. Periodic application (annual, 
biannual or as frequently as possible) 
aims to shed light on the decision- 
making of policymakers. 

At the project level, relevant data 
design in the intermediate steps of the 
project's implementation will inform 
progress and steer the project toward 
its proposed objectives, allowing for 
early problem detection and correction 
in learning. 

Technically precise periods (e.g. 
monthly, quarterly, biannually) may 
be formally set for development of the 
monitoring phase, according to spe- 
cific project characteristics, but tech- 
nological models may also be estab- 
lished enabling permanent feedback 
to project administrators in the form 


of pertinent information for monitoring 
and decision-making. 

3. Evaluation 

This process involves comprehensive 
review of a project, its achievements, 
progress and difficulties, and estab- 
lishes its impact vis-a-vis proposed 
objectives. Evaluation is conducted 
at project completion or at the end of 
a given phase of the project’s imple- 
mentation, and its purpose is to meas- 
ure actions and the strategy proposed 
against the results obtained, and to 
monitor its relationship with and impact 
on system indicators. 

Along these lines, impact made on all 
areas, processes and products must 
be taken into consideration and not 
only the ones where the project has 
implemented actions. 

Evaluation is a process that is crucial 
to every project and should be consid- 
ered an essential component at the 
outset of project design. Whenever 
possible, efforts should be made to 
have evaluation conducted by an 
external entity unassociated with the 
project’s direct or indirect executors, 
to achieve objectivity and impartial- 
ity. Whenever possible, experimen- 
tal evaluation methods should be 
favoured to complement other data 
sources to produce more solid, reli- 
able results. 

3. Indicators for ICT 
in education 

Application of the conceptual frame- 
work on a set of indicators is proposed 
as an exercise to facilitate a compre- 
hensive view at the system level (sub- 
national, national, regional or global) 
and at the project level as well. 
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Regarding use in monitoring systems, 
we propose creating an index based 
on a set of indicators to help describe 
the respective system. When apply- 
ing indicators at the project level, this 
set of indicators lends support to and 
organises the project evaluation proc- 
ess, but in no case is it completely 
exhaustive, since this process involves 
many other variables. 

For purposes of organising the indica- 
tors and associating them to the pro- 
posal framework, we have considered 
the need for input, process and impact 
indicators, depending on the data type 
you want to describe and its scope 
of application. Nonetheless, process 
indicators are applied exclusively at 
the project level and refer specifically 
to the components that each project 


proposes to develop; consequently, it 
is defined ad hoc. 

The methodological proposal for 
applying the indicators in the context 
of this conceptual framework and its 
associated indicators is comprised of 
five instances: 

7 . System index 

At the systems level, the IDB pro- 
posal is to consider all or the greatest 
number of indicators possible from 
among those proposed, in order to 
achieve the most complete view pos- 
sible of the development status for the 
incorporation of ICT into education. 

This set of indicators, to the extent that 
it is possible to obtain complete, up-to- 
date information, allows us to create 
one or more indices accounting for 
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the status of progress in the incorpo- 
ration of ICT into education, allowing 
us to determine the system’s phase of 
development and the areas in which it 
is more or less advanced. 

2. Selection of actions (policy) 

Based on the data provided by the indi- 
cators, policymakers can make better 
informed, more complete decisions 
that place a high priority on achieving 
specific impacts. 

3. Selection of relevant indicators 

At the project level, it will not be fea- 
sible for all defined input indicators to 
be modified by an ICT for education 
project. Therefore, at the outset of a 
project of this type, the first task to 
undertake would be to establish which 
of the proposed set of indicators are 
feasible for this initiative to impact. 
Furthermore, process indicators are 
created that will enable follow-up and 
monitoring of project development. 

Aside from the above, the methodo- 
logical proposal suggests that all input 
indicators must be measured each 
time or taken into consideration. This 
is done for two reasons. The first is 
because a project may, in practice, pro- 
duce impacts unforeseen in its original 
design, and it is advantageous to be 
able to ascertain and quantify them. 
The second reason has a systemic 
or public policy aspect to it: becoming 
acquainted with all of a country’s edu- 
cational indicators will afford a project 
executor a broader view of the global 
impact of any country’s different edu- 
cation projects and of its status at dif- 
ferent points in time. 

4. Follow-up and monitoring 
According to the methodological pro- 
posal at the system level, data gath- 


ering is to be carried out periodically 
at regular intervals, depending on the 
availability of data at each level: 

• before the project begins: building 
the baseline; 

• mid-term measurement: data gath- 
ering at the halfway point, while a 
project is being implemented. Allows 
you to determine impacts over the 
medium term and to take steps if 
necessary; 

end of project measurement: gath- 
ering information upon completion 
of the intervention; facilitates quan- 
tification of changes in indicators 
during the project implementation 
period. At this point, the status of all 
the input indicators is ascertained. 
These indicators provide informa- 
tion about the impact attributable 
to the project and about changes 
observed in the overall status of the 
system undergoing intervention. 

A fourth instance of data gathering is 
advisable, whenever possible: 

• follow-up measurement: gathering 
information one or two periods after 
the respective project is completed. 
This allows evaluation of the status 
of the situation over the medium 
term, after the project has ended. At 
this point, drops may be observed 
in some indicators due to lack of 
funding for recurring expenses, for 
example. 

Process indicators required for the 
project to report should also be 
defined. Reports on these indicators 
will be of utmost value to the project 
executor because this facilitates rigor- 
ous monitoring of project implementa- 
tion and provides the opportunity to 
make suggestions and, if necessary, 
to propose remedial measures. 
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At the outset of the project, it is advis- 
able to agree on a timetable for sub- 
mitting reports on these indicators. 
Perhaps not all of these indicators will 
be relevant to all of the processes. This 
means agreement must be reached 
among the parties regarding which 
indicators will be used for each project 
management plan and what reporting 
intervals will be observed. 

5. Impact evaluation 

The final evaluation of a project may 
take into consideration a broad set of 
tools, models and indicators to report 
on results. According to the proposal 
presented herein, we suggest taking 
into account how project results have 
enabled modification of indicators of 
the system where they were intro- 
duced, in terms of impact. These indi- 
cators were established in the defini- 
tion of the general indicators and in 
the selection of specific indicators rel- 
evant to project action. 

In this way, definition of the indicator 
allows us to set goals for the project, 
which under the same terms of the 
indicator it proposes to change. 
Therefore, for each relevant indicator, 
the project impact evaluation presents 
its respective status before the 
intervention, the status targeted by the 
intervention (goal) and the percentage 
of the goal achieved. 

4. Conclusions 
and next steps 

We have worked on this conceptual 
framework and indicators proposal in 
the belief that information and com- 
munication technologies (ICT) can 
indeed make an important contribution 
to improving the quality of education, 
but they demand a much more rigor- 
ous, comprehensive incorporation into 


educational systems and the support 
of empirical evidence on how to opti- 
mally capitalise on ICT potentials. 

ICT alone will not make the difference. 
We are confident that no technological 
device will solve the enormous chal- 
lenges facing the education systems 
seeking to meet today’s demands. 
We are not facing a technological 
challenge, but an educational chal- 
lenge ( 22 ). We know that training 
people, a country’s human capital, is a 
complex process involving a myriad of 
variables with which ICT must interact 
dynamically to produce the changes 
required. 

We acknowledge that we are facing a 
challenge that is both vast and new, 
but which also changes at speeds 
heretofore unseen. Therefore, we 
expect this proposal will undergo 
continual revisions, adjustments and 
reformulations. We present it with the 
humility of an individual who explores 
unknown lands without the benefit of 
certainties or necessary tools, but with 
the urgency of having to move forward 
with determination. 

Currently we are preparing the pro- 
posal for indicators that reflect and 
complement the scale proposed in 
this conceptual framework. To accom- 


( 22 ) ‘Today’s classroom teachers need to be 
prepared to provide technology-supported 
learning opportunities for their students. Being 
prepared to use technology and knowing how 
that technology can support student learning 
have become integral skills in every teacher’s 
professional repertoire. Teachers need to 
be prepared to empower students with the 
advantages technology can bring. Schools and 
classrooms, both real and virtual, must have 
teachers who are equipped with technology 
resources and skills and who can effectively 
teach the necessary subject matter content while 
incorporating technology concepts and skills.’ 
(Unesco, 2008) 
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plish this task, we are considering a 
very important proposal that Unesco 
UIS has already developed including 
over 50 indicators, which we are com- 
plementing with additional indicators 
covering all the areas proposed. 

We are making this seminal work 
available to those who wish to col- 
laborate in its continual improvement 
and to the development of tools that 


allow us to apply it to our specific con- 
texts. We are still striving to improve it, 
in collaboration with experts and other 
agencies and international organisa- 
tions. It is now being implemented in 
currently operating bank-supported 
projects in Latin America and the 
Caribbean for the purpose of aligning 
definitions, specifying and testing indi- 
cators and building new instruments 
for its implementation. 
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A conceptual framework 
for benchmarking the use and assessing 
the impact of digital learning resources 
in school education 

Benat Bilbao-Osorio and Francesc Pedro 

OECD Centre for Educational Research and Innovation 

The comparative study of information and communication technologies (ICT) in 
school education has primarily focused on investments in infrastructure, equip- 
ment and the resulting ratios per pupil, as well as on In-service teacher training 
and, lately, on the incentives and barriers for classroom use. Less attention has 
been paid to the development and publication of digital learning resources (DLR) 
as a mean to increase the added value that ICT can bring about for teaching and 
learning. In some countries, governments have started to subsidise programmes, 
repositories and networks focusing on DLR. However, until now, little empirical 
evidence has existed on the dimensions and impact of these policies, both on 
their capacity to foster the development of DLR and on their final effects on the 
teaching and learning processes. 

The OECD Centre for Educational Research and Innovation (CERI) has recently 
completed a project intended to bridge this knowledge gap by reviewing and eval- 
uating the process of innovation involved in policies and public as well as private 
initiatives designed to promote the development, distribution and use of DLR for 
the school sector. Among its final outputs j 23 ), this project includes the delivery of 
a conceptual framework for the creation of a system of indicators related to the 
development, use and effects of DLR. 

This chapter presents the resulting initial proposal. It aims at shedding more em- 
pirical light on the theoretical and policy debate about the effects of technology- 
enhanced learning in schools. In this respect, the chapter sets the scene for the 
ongoing policy debate and then discusses the lack of empirical evidence. Then it 
outlines the objectives of the CERI proposal and describes its main components. 
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The policy debate 

Investment in ICT as a means to raise 
educational standards and create 
opportunities for every child has 
attracted attention from both policy- 
makers and academic researchers. 
Based on the belief that ICT could 
make learning much more efficient, 
and encouraged by the findings that 
new technologies had accounted for 
much of the productivity gains in the 
workplace during the advent of the 
new economy (Lindbeck and Snower, 
2000), many governments explored 
the role that ICT could play for edu- 
cation, although this interest started 
to fade away in some countries after 
the 2001 technology crisis and a grow- 
ing awareness that the investments 
seemed not to produce any relevant 
accountable effects. 

Nevertheless, the benefits of the use 
of ICT have been largely echoed 
among policymakers and claims like 
‘ICT can make a significant contribu- 
tion to teaching and learning across all 
subjects and ages, inside and outside 
the curriculum’ (UK DfES, 2003) have 
been common. Four main sources of 
educational benefits have been identi- 
fied with the use of ICT: (i) preparation 
of students for the knowledge econ- 
omy; (ii) improvement of educational 
performance; (iii) struggle against 
the ‘digital divide' between students; 
(iv) improved quality in the teaching 
and learning processes, allowing for 
a customisation of the educational 
processes. 

Based on these expected benefits, sig- 
nificant government investments have 
been made available. Between 1998 
and 2002, ICT expenditure in the UK 
almost doubled in secondary schools 
and multiplied by three in primary 
schools. Equally, 10 years ago, the 


OECD already reported that education 
policymakers saw enormous potential 
for ICT to transform education. In 1999, 
the limited available data on trends in 
ICT investment and use were headed 
up sharply (OECD, 1999). Around that 
time, an OECD conference warned 
about the urgency of ‘bridging the 
digital divide’ (OECD, 2000). In 2004, 
PISA data confirmed the exponential 
growth in the presence of ICT in educa- 
tion (OECD, 2004). In just three years, 
between 2000 and 2003, student-per- 
computer ratios dropped by more than 
half in most countries (and by a factor of 
4 to 5 in those that were lagging). While 
less than a third of secondary schools 
had Internet access in 1995, it was vir- 
tually universal by 2001 . Although there 
are no internationally comparable data 
on current expenditure on educational 
ICT hardware and software, there are 
signs of unmet demand for additional 
investment, particularly in the areas of 
hardware upgrading and availability of 
digital content or learning resources. 
According to the most recent PISA 
data, a lack of adequate computer soft- 
ware for instruction is cited by school 
principals as an important hindrance to 
science instruction (OECD, 2007) 

However, there seems to be little 
empirical evidence ( 24 ) about the 
final benefits associated with these 
investments in terms of use of ICT in 
schools and their impacts throughout 
the educational system, and claims 
of ‘unfulfilled promises’ have opened 
an academic and policy debate about 
whether the considerable investment 
in ICT pays off in any obvious way. 


( 24 ) Recently a number of studies have aimed at 
analysing the impacts of ICT in education. The 
analytical works at SITES, E-Nordic Learning, 
Becta or the OECD’s PISA reports (based on 
2003 and 2006 results) are the main experiences 
in the field. 
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Lack of empirical evidence 
about the effects of DLR, 
and ICT more broadly, 
on education 

Computers in education are gener- 
ally used in two broad contexts: (i) 
to provide computer skills training, 
which teaches students how to use 
computers; (ii) to provide technol- 
ogy enhanced learning (TEL), which 
uses computers to enhance teach- 
ing and learning methods, strategies 
and activities in the whole curriculum. 
While there is a clear case for the 
use of ICT for enhancing the compu- 
ter skills of students, the role of TEL 
is more controversial (Machin et al., 
2006). There is neither a strong and 
well-developed theoretical case nor 
much empirical evidence supporting 
the expected benefits accruing from 
the use of ICT in schools, as different 
studies find mixed results (Kirkpatrick 
and Cuban, 1998). 

Apparently, there seems to be no con- 
clusive evidence. On the one hand, 
studies carried out by Becta (2002) 
and Machin et al. (2006) find a posi- 
tive effect on the use of ICT and edu- 
cational attainment, and, on the other 
hand, the research carried out by 
Fuchs and Woessman (2004), Leuven 
et al. (2004) or Goolsbee and Guryan 
(2002) find no real positive effect 
between the use of ICT and educa- 
tional results once other factors, such 
as school characteristics or socio- 
economic background, are taken into 
account. There is a generalised belief 
that, overall, the ‘no significant differ- 
ence’ phenomenon, documented on 
many occasions in the case of dis- 
tance learning, also emerges in school 
education. According to this, there is 
insufficient evidence to affirm either 
the superiority or inferiority of ICT- 


rich methodologies. This would seem 
to be the outcome of the two system- 
atic reviews of literature conducted 
recently, which conclude that ‘in gen- 
eral and despite thousands of studies 
about the impact of ICT use on stu- 
dent attainment, it is difficult to meas- 
ure and remains reasonably open to 
debate' (/'nfoDev, 2005), and also that 
‘some studies reveal a positive cor- 
relation between the availability of 
computer access or computer use and 
attainment, others reveal a negative 
correlation, whilst yet others indicate 
no correlation whatsoever between 
the two’ (Kozma, 2006). 

Experiments can only attempt to deter- 
mine how effective ICT is in teach- 
ing specific school subjects, due to 
the multitude of compartmentalised 
methodologies to be found in a single 
school, and even in lines or different 
groups of students studying the same 
subject, albeit with different teach- 
ers. Consequently, the experiments 
designed to date compare the educa- 
tional attainment of a group of students 
using an ICT-rich teaching methodol- 
ogy with the achievement of another 
group with similar characteristics being 
taught using traditional methods. 

However, an in-depth analysis of the 
available knowledge base shows that 
school attainment only improves if cer- 
tain pedagogical conditions are met. 
This is the conclusion reached by Kulik 
(2003), who used the measurement 
of the effects found by eight different 
meta-analyses covering 335 studies 
before 1990 and 61 controlled experi- 
ments whose outcomes were pub- 
lished after 1990. Most of the studies 
carried out in the 1 990s concluded that 
stimulation programmes have positive 
effects when used to enhance reading 
and writing capabilities and that, albeit 
less frequently, they have a clearly 
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positive effect on maths and natural 
and social sciences. Indeed, ‘simply 
giving students’ greater access to both 
computers and Internet resources 
often results in improved writing skills’. 
The assessments of primary school 
pupils using tutorials to improve their 
writing increased significantly in this 
field. Even very young primary school 
pupils using computers to write their 
own stories ended up improving their 
marks in reading. In short, there is a 
positive correlation between the fre- 
quent use of word processors and 
improved writing-related capabilities. 

Much less attention has been paid 
both by researchers and policymakers 
to the actual determinants of ICT use 
in school and their impacts in different 
dimensions of the educational system. 
For a long time, as noted above, ICT 
investments have been channelled 
towards the construction of an ICT 
infrastructure in schools, and most 
available resources have been devoted 
to the acquisition of ICT equipment, 
i.e. computers, and of Internet access 
connections, e.g. broadband net- 
works. While this investment is a clear 
pre-requisite to foster the use of ICT in 
schools, it can also be regarded as a 
necessary but not sufficient condition 
to assure its use, if other factors are 
not simultaneously born in mind. More 
precisely, factors such as the compe- 
tences and attitudes of teachers to 
use ICT or the availability of DLR have 
also been identified as key factors to 
explain the degree of use of ICT by 
teachers and students. 

While teachers’ attitudes and compe- 
tences in respect of ICT have been 
widely recognised as a key factor 
(Williams et al., 1998) and signifi- 
cant public investments have aimed 
at enhancing these competences, 
much less attention has been paid to 


the development of DLR and to the 
development of content production. 
Although many big private publishing 
companies have entered the market 
of developing DLR and have acknowl- 
edged their potential, until recently 
they have regarded this market as 
unattractive and major investments 
have not been made. A possible 
explanation for this may lie in the 
role that private publishers play in the 
development of school content, either 
in analog or digital form. Commercial 
publishers have traditionally played a 
key role in developing and distributing 
printed learning material. However, 
when it comes to DLR, they seem to 
find that the market may not be ready 
to use this type of resource yet, mainly 
due to the lack of infrastructure, teach- 
ers’ skills or cultural factors. Therefore, 
they may lack the necessary incentive 
to develop this kind of material. At the 
same time, the lack of readily avail- 
able DLR of sufficient quality can also 
affect the motivation and attitudes of 
teachers towards DLR and ICT more 
broadly, and the need to invest in ICT 
infrastructures. On the whole, a vicious 
circle appears when the lack of signifi- 
cant teacher demand proves a disin- 
centive to publishers’ offers, which in 
turn affects demand negatively, and 
where all the determinants are closely 
intertwined. 

In addition to private publishers, stu- 
dents and teachers have also started 
producing DLR by themselves, partly 
along the lines and rationale which are 
successfully inspiring the production 
and use of open educational materi- 
als in higher education ( 25 ). There has 
been a shift in the paradigm where 
both teachers and students were only 


( 2S ) See CERI-OECD, Giving knowledge for free: 
open educational resources in higher education. 
(Paris, OECD, 2007). 
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users of learning material, and they 
are now also producing content mate- 
rial which they exchange among them- 
selves and that is regarded by their 
peers as ‘very important’. The mate- 
rial of these ‘user-producer’ teachers 
and students is increasingly important 
and will continue to be so as Web 2.0 
applications become more available 
generally. However, until now, its study 
has also been somehow neglected in 
traditional studies. 

Objectives of the 
conceptual framework 

The overall aim of this proposal is to 
bridge this analytical gap in the study 
of DLR and deliver a conceptual 
framework for developing indicators 
that could trace and benchmark the 
development, use and effects of DLR. 

More precisely, the objectives of this 
proposal are as follows. 

1. To provide a holistic conceptual 
framework for the development of 
these indicators. This model would 
map the different factors affect- 
ing the development and use of 
digital learning resources, and 
their impacts on the educational 
system. 

2. To define and construct a number 
of key indicators that would allow 
comparing and benchmarking 
across different countries of the 
progress in the production, avail- 
ability, use and impacts of DLR in 
schools. 

3. To identify existing relevant 
sources and collect the available 
data. Based on the different fac- 
tors described on the conceptual 
framework, to identify what data 
are already available in different 


data sources and the possibility (or 
not) of linking different datasets. 

4. To highlight possible options to 
generate the missing data. As a 
result of the analysis of the data 
already available, data gaps will be 
identified and different strategies 
and tools to develop the required 
data will be suggested. 

Definition 

While there is a clear and practical 
interest to track the availability and use 
of DLR, there is an even greater inter- 
est in understanding the causes driv- 
ing the development and use of DLR, 
and the impacts they generate on the 
teaching and learning processes, 
because the lessons learnt can be 
used to refine our understanding of the 
incentives and barriers regarding the 
broader use of ICT to enhance school 
education. An analytical framework 
capable of identifying and explaining 
these factors, their interrelations and 
their impacts would allow analysts to 
enhance their knowledge about the 
use of DLR and ICT more broadly, and 
to provide evidence-based policy rec- 
ommendations for policymakers. 

However, at the moment, the lack of 
a holistic conceptual framework that 
takes into account all the intervening 
factors and their possible interrelation- 
ships, and the lack of available data 
have prevented the development of 
more robust results allowing to moni- 
tor and evaluate the role that different 
sources of ICT investment, including 
investments in DLR, play in the use 
of ICT and in the teaching and learn- 
ing processes and the educational 
attainment of students. This lack of 
empirical evidence has also affected 
the necessary political support for any 
further investments and has increased 
the feeling among stakeholders of 
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‘unfulfilled promises’ related to the use 
of ICT in the educational system. 

In light of the information gathered in 
the OECD project on DLR during the 
interviews conducted with a number 
of stakeholders (i.e. departments of 
education, teachers, headmasters, 
students, local and regional govern- 
ments and publishers) and a review of 
the existing literature on comparative 
research and recent practices, an ana- 
lytical framework is proposed below. 
This framework aims to account for 
the factors affecting the development, 
use and impacts of DLR, as well as 
for the complexity of the interrelation- 
ships between these factors. Figure 1 
presents a visual representation of 
this framework. 

The proposed model presents a 
number of investment measures on 
the left-hand side of the chart that 
are interrelated. Each of these invest- 


ments produces a specific output in the 
form of available computers or Internet 
access (for the case of ICT infra- 
structure), digital learning resources 
or enhanced teachers' ICT compe- 
tences. The combination of these out- 
puts would influence the actual use 
of DLR and ICT more broadly, in a 
particular moment in the educational 
system. However, rather than claiming 
a linear and causal relationship, the 
model intends to reflect the complex 
nature of the interaction between each 
of these factors and the actual use of 
ICT/DLR. For instance, higher levels 
of ICT/DLR use could also stimulate 
higher levels of ICT/DLR investments. 

In addition to these three main direct 
investment variables, a number of 
‘environmental factors' would also 
affect the levels of DLR/ICT use and 
therefore should be included in the 
model. These variables relate to the 



Figure 1: Analytical framework for assessing the development, use and impacts of DLR 
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overall ICT environment in the coun- 
try that may push for or against the 
use of ICT in society in general, and 
in the educational system in particular. 
Particular attention has to be paid to 
the fact that very different factors can 
be brought into the picture. The degree 
of public policy influence on these fac- 
tors could largely differ both in scope 
and impact depending on the nature of 
these factors. Teachers’ commitment 
to the use of ITC in classes, for exam- 
ple, is a key variable that affects the 
final use of DLR or ICT in schools, and 
that would be the result of a mix of fac- 
tors such as policies to promote ICT 
in schools and teachers' attitudes and 
convictions regarding the role of ICT in 
the teaching and learning processes. 
Pupils’ expectations would be another 
variable that could significantly affect 
the use of DLR and ICT and that 
could be far from being affected by 
public intervention. These factors are 
somehow the ‘soil’ where the DLR/ 
ICT investments are ‘seeded’ and that 
could be a determinant in obtaining 
the desired ‘fruit’. 

As a result, policymakers are con- 
fronted with a policy dilemma in terms 
of what to do: invest in infrastructure, 
DLR, teaching competences (in which 
ones, and how much?) and/or in 
improving the ICT environment (how, 
and how much?) in order to obtain the 
desired results in terms of enhancing 
the ICT/DLR use. 

Finally, the model suggests that the 
use of ICT/DLR could have a final 
impact on the educational system by 
allowing students to achieve higher 
educational attainment, developing 
stronger digital competences and 
improving the perceived satisfaction 
in the teaching and learning proc- 
esses. However, as happened previ- 
ously, the relationship between the 


variables is not unidirectional, and 
therefore higher levels of technologi- 
cal competences, better academic 
performances or higher levels of sat- 
isfaction in the teaching and learning 
processes could also influence higher 
ICT/DLR uses, triggering a virtuous 
upwards circle that would move within 
the whole model. 

The relationships between the differ- 
ent variables in this model are hypo- 
thetical and their existence (or not) 
should be investigated empirically, 
should data become available. 

The variables 

The model described above presents 
a number of variables and hypotheti- 
cal relationships between the vari- 
ables that would need to be tested. 
This section briefly presents the dif- 
ferent variables. As presented, this 
section only identifies the variables 
and provides some initial suggestions 
for their definition and measurement. 
The difference in scope of these defi- 
nitions would therefore affect the type 
of data that would be required. These 
variables, classified according to their 
nature and role in the proposed model, 
are as follows. 

1. Direct investment variables: 

These are the different sources of 
investment where a clear connec- 
tion between the initial investment 
and the actual results accruing from 
them can be identified. The model 
identifies three investment types, 
closely intertwined between them. 

• ICT infrastructure : This vari- 
able deals with the investment 
in equipment (computers, white- 
boards, laptops, projects) and 
network connections. A number 
of clear outputs can also be 
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observed as a direct result of 
these investments: the number 
of computers by students or 
the number of computers with 
(broadband) Internet connection 
by student are just a few exam- 
ples of this type of variable. 

• Digital learning resources (DLR): 
There have been many defini- 
tions of DLR. In this project ( 26 ), 
it has been pointed out that DLR 
‘can refer either to any resource 
used by teachers and students 
for the purpose of learning, or 
to only resources particularly 
designed to be used in learning 
settings. It is both a strength and 
a weakness of the former defini- 
tion that it is very general — it 
can refer to anything from a stone 
or a feather, to Encyclopedia 
Britannica or advanced data- 
bases, as long as it is used for 
learning. The second definition is 
more limited and hence easier to 
use. But it excludes open learning 
resources like online newspaper 
articles, most computer games, 
and applications such as Google 
Earth’. Moreover, it would not 
take into account the production 
of DLR carried out by individual 
teachers and shared within a 
closed system or intranet exclu- 
sively. As a result, it is important 
to note that this definition and 
measurement would be a stricter 
approximation of the overall DLR 
concept and therefore any con- 
clusions about the availability 
and role of DLR should be han- 
dled very carefully. 

• Teachers’ ICT competences: This 
variable relates to those invest- 
ments aiming at making the 


( 26 ) Please refer to the full OECD report Beyond 
Textbooks. Digital Learning Resources in the Nordic 
Countries for a more thorough definition of DLR. 


teachers more competent and 
eventually having a positive atti- 
tude towards ICT and using ICT 
in school. The input investment 
would be the resources devoted 
towards teacher training and ICT. 
The output measure, however, 
could differ and allow for differ- 
ent definitions and measures. On 
the one hand, an easy and direct 
measure could be the number of 
teachers trained in the system. 
On the other hand, a more com- 
plex measure could relate to the 
attitudes and changes in atti- 
tudes of the trained teachers 
towards the use of ICT/DLR. 

2. Outcomes: An intermediate out- 
come can be linked and traced 
back to the initial investment vari- 
ables, but can also be influenced 
by some external factors. 

• Use of ICT/DLR: The amount 
and nature of the different uses 
of DLR and ICT. This broad vari- 
able could be broken down into 
different categories and create 
a typology of different types of 
ICT/DLR uses according to the 
different categories of DLR, for 
example. Equally, a classification 
of the use by subject and class 
group would also provide more 
information that could be useful 
when analysing its relationship 
with the investment variables. 

3. Impacts: These are the final objec- 
tive that the initial investments aim 
at. The model identifies two main 
types of possible impacts. 

• Student performance: The use 
of ICT and DLR could have an 
impact on the student’s per- 
formance that could go in two 
directions: 
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— Development of ICT compe- 
tences (or ‘21st century com- 
petences’): The definition 
of ICT competences could 
be restricted to the effective 
use of the ICT infrastruc- 
ture, i.e. use of a computer 
or the Internet, or it could 
have a broader scope, where 
students would be able to 
use, search, understand 
and even produce different 
content in a digital support 
in order to obtain or show a 
better understanding of par- 
ticular subjects. In the latter, 
specific definitions of com- 
petences should be devel- 
oped and appropriate tests 
should be put in place in 
order to measure and evalu- 
ate the achievement of these 
competencies. 

— Academic performance in 
basic subjects'. The use of ICT 
in learning different subjects 
could have an impact in the 
actual academic attainment 
of students in these differ- 
ent subjects. Analysing these 
results and comparing them 
before and after the use of 
ICT/DLR would be important 
to establish any causal rela- 
tionship between the two. 

• Improved or new teaching and 
learning process: The use of 
DLR and ICT could also improve 
or bring about new processes 
of both teaching and learning, 
making it more interesting for stu- 
dents and teachers, and improv- 
ing the communication between 
the different stakeholders. 
Having an objective’ measure of 
‘improved’ process could be very 
difficult as it would require a clear 
definition and measurement of 
all the different aspects affect- 


ing this process, including the 
always fuzzy concept of quality. 
However, a ‘subjective’ measure- 
ment of the changes in the proc- 
ess by the different stakeholders 
could be a way to get around this 
initial difficulty. 

4. Environmental factors: These 
variables, although they cannot be 
directly controlled by direct govern- 
ment investment, have a very clear 
impact in the capacity of the direct 
investments to achieve the desired 
results. They are 'the soil’ where 
the different investments (‘the 
seeds’) are planted. 

Teachers’ commitment to ICT: 
The teachers’ commitment and 
determination to use ICT and 
DLR in their schools is one key 
variable that may explain differ- 
ences in the levels of investment 
in schools and also in the actual 
use of ICT/DLR by the teachers. 
This is particularly true in decen- 
tralised systems, where teach- 
ers count on a large degree of 
autonomy. Also, research has 
shown the relevance of leader- 
ship in schools in this domain. 

• Socioeconomic factors: Socio- 
economic background, age and 
gender of students have been 
pointed out in the literature as 
being key factors that may influ- 
ence not only their learning 
expectations but also the degree 
and scope of the actual use of 
ICT/DLR (outcome variable), 
and also influence decisively the 
students’ educational attainment 
(impact variable). Therefore, any 
study that aims at drawing causal 
relationships between the vari- 
ables should take these factors 
into account. 
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• In addition to these variables, 
it is important to note that the 
model also identifies a very broad 
variable that somehow affects 
all the different variables in the 
model. 

• The overall ICT environment. This 
variable aims at explaining the 
overall societal attitude towards 
the use of ICT, not only in the edu- 
cational system, but more broadly 
in all aspects of life. This broad 
variable would include: 

— ICT responsiveness: ICT 

readiness and acceptance in 
the overall society influence 
the pressure and demand 
for the inclusion of ICT in the 
educational system as well as 
the attitudes of both teachers 
and students towards the use 
of ICT. Possible measures 
of this responsiveness could 
be the penetration of ICT in 
homes or in firms. 

— National curriculum: The 

inclusion of the obligation to 
use ICT/DLR in the national 
curricula, directly or indirectly 
(by way of mentioning them 
in the definition of expected 
pupils’ competences), may 
be a variable that may explain 
difference across countries in 
the use of ICT/DLR, and also 
may be a factor affecting the 
levels of ICT/DLR investments 
in the educational system. 

Conclusions and next steps 

Benchmarking DLR use and effects in 
education is not an easy task. To start 
with, in the complex context of a class- 
room, it is difficult to isolate the DLR 
contribution from other elements that 
intervene in the educational process. 
Access to quality DLR is certainly an 
enabler of better educational opportu- 


nities, but in itself access to DLR does 
not imply automatically granting better 
educational processes. How DLR is 
used, in the wider context of other 
intervening factors, is the critical vari- 
able — and not much is known about 
it yet. This points out the need for a 
clear understanding not only of the 
intervening factors but, in particular, of 
their interrelationships. 

CERI’s ongoing work in this domain 
is addressing some of these issues in 
close cooperation with other interna- 
tional agencies. The main objective is 
to enlarge the number and quality of 
indicators about access to and use of 
ICT in education. In so doing, the main 
activities that will be carried out are as 
follows. 

1. Redefinition and refinement 
of the model: A validation of the 
model should be carried out. More 
precisely, this activity would (re-) 
define and identify new factors, 
map the hypothetical relationships 
between the variables and revisit 
the scope of the model. This refine- 
ment of the model would allow 
building the necessary consensus 
in order to develop internationally 
agreed and comparable indicators. 

2. Redefinition of variables: Alter- 
native definitions for the variables 
are available, with differences in 
scope and nature. A commonly 
agreed redefinition of the variables 
would then be necessary. 

3. Evaluation of available data: 

Based on the agreed model, an 
evaluation of the existing data 
sources and the possibility to link 
different datasets in a coherent 
manner should be carried out. 
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4. Data needs assessment: Based 
on the agreed model and the data 
already available, a data needs 
assessment should be carried out. 
As mentioned in the definition of the 
variables of the conceptual model, 
the data needs can be defined in 


different levels of depth. The com- 
plexity and cost to obtain the data 
should match the utility and a con- 
sensus should be reached when 
defining the variables and devel- 
oping the necessary methods to 
obtain the required new data. 
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Abstract 

As technologies and contexts of their use increase, characterizations of 21st cen- 
tury skills have grown beyond operation of computer productivity tools to encom- 
pass individuals' use of the Internet, specialized software, and facility with hand- 
held and wireless devices. ‘New literacies’ have expanded to refer to expertise in 
the use of a range of digital media and information and communication technolo- 
gies exercised in academic and applied settings to solve a range of problems 
(Quellmalz & Haertel, 2008). This paper addresses: (1) distinguishing features of 
the multiple frameworks for ICT, 21st century skills, and new literacies; (2) alterna- 
tive assessment designs and prototype student assessments of new literacies, 

(3) evidence-centered design methods for establishing technical quality, and 

(4) features of coherent, balanced assessments of new literacies across class- 
room, district, state, national and international levels. 


Introduction 

Information and communication tech- 
nologies (ICT) permeate school, work, 
personal and civic activities. Their 
prevalence speaks to the centrality of 
these powerful, transformative tools in 
all walks of life. Policymakers through- 
out the world recognise the significance 
of technologies for economic, civic and 
global progress, along with the con- 
comitant need for coherent educational 
policies to promote and implement skills 
characterised as ‘new literacies', 21st 
century skills, information and commu- 
nication technology skills and techno- 
logical literacy (ISTE, 2007; Partnership 
for 21st Century Skills, 2005; Kozma, 
McGhee, Quellmalz, and Zalles, 2004). 


As technologies and contexts of their 
use increase, characterisations of 21st 
century skills have grown beyond oper- 
ation of computer productivity tools 
to encompass individuals’ use of the 
Internet, specialised software and facil- 
ity with handheld and wireless devices. 
‘New literacies’ have expanded to refer 
to expertise in the use of a range of 
digital media and information and com- 
munication technologies exercised in 
academic and applied settings to solve 
a range of problems (Quellmalz and 
Haertel, 2008). 

Technologies are increasingly rec- 
ognised as transforming schooling 
as a result of their capacity to extend 
students’ opportunities to access 
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rich repositories of knowledge and to 
engage in deep, extended problem 
solving. Large-scale national and inter- 
national studies are providing evidence 
that technologies are truly changing 
and improving schools by enriching cur- 
ricula, tailoring learning environments, 
offering opportunities for embedding 
assessment within instruction and pro- 
viding collaborative tools to connect 
students, teachers and experts locally 
and globally (Kozma, 2003; Law, 
Pelgrum and Plomp, 2008). Despite 
the pervasiveness of technology, there 
are few traditional large-scale tests 
or curriculum-embedded, formative 
measures that directly measure new lit- 
eracies (Burns and Ungerleider, 2002; 
Quellmalz and Kozma, 2003). 

The quest for tests of students’ profi- 
ciencies with these 21st century skills 
is hindered by a number of persistent 
issues. There are myriad definitions of 
information and communication tech- 
nologies and technological literacy 
knowledge and skills. The contexts in 
which ICT should be taught and tested 
vary widely. The extent to which the 
knowledge and skills about technolo- 
gies to be used within a domain-based 
problem or context can be distinguished 
from the domain-specific knowledge 
and skills required is ambiguous 
(Bennett, Jenkings, Persky and Weiss, 
2003; Quellmalz and Kozma, 2003). 
Methods for designing 21st century 
assessments and for documenting their 
technical quality have not been widely 
used. Finally, a critical issue facing 
the promotion of 21st century learn- 
ing is that assessments of ICT should 
be coherent across levels of educa- 
tional systems (Pellegrino et al., 2001). 
Coherence must start with common or 
overlapping definitions of the knowl- 
edge and skills to be assessed as 
new literacies. If the designs of inter- 
national, national, state and classroom 


level tests of new literacies are not 
aligned and articulated, the assess- 
ment systems will not be balanced and 
the validity of inferences about student 
performance will be compromised. 

'New literacy' assessments 

Currently, there are multiple frame- 
works for assessing technology use 
and 21st century critical thinking and 
problem-solving processes. In one 
view, ICT assessment is oftechnology, 
such as the international computer 
driving licence and technology profi- 
ciency tests in some states in the USA. 
These tests measure the facts and pro- 
cedures needed to operate common 
Internet and productivity tools, while 
the content or the academic or applied 
problem and context are deliberately 
selected to be familiar background 
knowledge (Venezky and Davis, 2002; 
Crawford and Toyama, 2002). The 
cognitive processes addressed in 21st 
century skills frameworks such as prob- 
lem solving, communication, collabora- 
tion, innovation and digital citizenship 
are not targeted by tests of technology 
operations. 

In a second view, ICT and 21st century 
frameworks emphasise learning with 
technology by presenting test problems 
and items that integrate measurement 
oftechnology operations, strategic use 
of technology tools to solve problems 
and subject matter knowledge and 
processes through carefully designed 
sets of tasks and items related to com- 
plex academic and real world prob- 
lems. This is the most prevalent view in 
21st century ICT frameworks. 

In a third view, testing is implemented 
by technology. Assessments by tech- 
nology simply use technical infrastruc- 
tures to deliver and score tests that 
are designed to measure other content 
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and skills in subjects such as maths 
and reading. These test designs aim 
to reduce or eliminate the demands 
of the technology, treating it as an 
irrelevant construct. Equivalence of 
paper-based and technology-based 
forms is the goal. Technology-based 
tests are increasing rapidly in large- 
scale state, national and international 
testing, where technology is being 
embraced as a means to reduce the 
costs and logistics of assessment func- 
tions such as test delivery, scoring and 
reporting. Technology-based tests typi- 
cally assume that supportive technol- 
ogy tools such as calculators or word 
processors are irrelevant to the content 
constructs being tested and therefore 
not to be measured separately. Since 
these types of testing programs seek 
comparability of paper and online 
tests, the tests tend to present static 
stimuli and use traditional constructed- 
response and selected-response item 
formats. For the most part, these con- 
ventional online tests remain limited 
to measuring knowledge and skills 
that can be easily assessed on paper. 
Consequently, they do not take advan- 
tage of technologies that can measure 
more complex knowledge structures 
and extended inquiry and problem solv- 
ing included in 21st century ICT frame- 
works. In short, a technology delivered 
and scored test of traditional subjects 
is not an assessment of 21st century 
ICT skills and should not be confused 
as one. 

This paper focuses on assessments 
of technology and assessments with 
technology, not assessments by tech- 
nology. It addresses: (i) distinguishing 
features of the multiple frameworks 
for ICT, 21st century skills and new 
literacies; (ii) alternative assessment 
designs and prototype student assess- 
ments of new literacies; (iii) evidence- 
centered design methods for establish- 


ing technical quality, and (iv) features 
of coherent, balanced assessments of 
new literacies across classroom, dis- 
trict, state, national and international 
levels. 

Features of new literacy 
assessment frameworks 

Different specifications of knowl- 
edgeand skills: Numerous frameworks 
have been developed by international, 
national, state and professional organi- 
sations to specify the important char- 
acteristics of new technology-based 
literacies, variously named ICT literacy, 
21st century skills and technological lit- 
eracy. These frameworks differ in the 
range of technologies included, the 
types of processes assessed in their 
use and the types of contexts of prob- 
lems in which the technologies will be 
applied. The frameworks differ in their 
focus on common Internet and produc- 
tivity tools such as browsers, graphing 
tools, word processors and presentation 
tools and inclusion of more advanced, 
specialised tools such as visualisa- 
tions, simulations and domain-specific 
datasets and software. The frameworks 
differ in their relative emphases on the 
operation of technology tools in con- 
trast to the use of the tools along with 
21st century skills for solving problems 
and achieving goals in practical or aca- 
demic domains. Common processes 
often include accessing, organising, 
representing, analysing, evaluating, 
synthesising, communicating and col- 
laborating (ISTE, 2007; Partnership for 
21st Century Skills, 2005). The forth- 
coming 2012 framework for the US 
National Assessment of Educational 
Progress (NAEP) for Technological 
Literacy has expanded the conceptu- 
alisation of the kinds of technologies 
and contexts of their use even further 
by integrating engineering design and 
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ICT technological literacy frameworks. 
The NAEP technological literacy frame- 
work specifies three major assessment 
areas: technology and society, engi- 
neering design and systems and ICT 
(see http://naeptech2012.org). The 
addition of engineering design to ICT 
frameworks incorporates knowledge 
and skills about how technologies are 
developed as well as how they are used. 
The 2012 NAEP technological literacy 
framework, which will shape the next 
decade of designs of the assessments 
that serve as ‘the nation’s report card’, 
deliberately specifies a wider range of 
technology products and processes 
than those called out in ICT frame- 
works. Technologies in the designed 
world include those in contexts such as 
transportation, energy, agriculture and 
health, as well as information and com- 
munication technologies (ITEA, 2000). 

Specifications for assessment vs. 
curriculum: A critical issue in the 
assessment of the new technological 
literacies is the distinction between 
curriculum and assessment frame- 
works. The standards specified by 
ISTE, ITEA and national frameworks 
set goals for promoting technology 
understanding and use. These stand- 
ards aim to shape curriculum, instruc- 
tion and assessment. In contrast, a 
national or international assessment 
framework may limit the content and 
skills specified in the frameworkto what 
can be directly tested and reported in 
large-scale, on-demand assessments. 
Thus, extended projects, collaboration 
and teamwork or creativity are unlikely 
to be tested in systematic, replicable 
ways on the large-scale tests, but can 
be promoted and potentially assessed 
at the classroom level. 

The role of domain knowledge: 

Another issue is the role of knowledge 
about topics and contexts required 


to complete tasks and items using 
technology. Background knowledge 
from life experience will present task 
demands different from tasks requiring 
knowledgefromacademicsubjects. For 
assessments of students’ ability to use 
technologies in a range of academic 
and practical problems, assessment 
frameworks must be explicit about the 
areas, complexity and familiarity of 
content in assessment tasks or items 
and if that knowledge will be scored in 
addition to processes and operations. 

The next section describes a coordi- 
nated assessment framework devel- 
oped in an international project that 
aimed to provide a cross-cutting set 
of knowledge and skills that could be 
used to test ICT literacy in academic 
or applied contexts. 

A coordinated framework 
for the design of ICT 
assessment 

The purpose of the project to develop 
a coordinated ICT assessment frame- 
work was to integrate measurement 
of technology use, ICT strategies and 
subject matter. Development of the 
coordinated ICT assessment frame- 
work was one component of a three- 
year study funded by the National 
Science Foundation (NSF) (Quellmalz 
and Kozma, 2003). The project goals 
were to develop a coordinated ICT 
framework and to design ICT per- 
formance assessments that could 
be administered as a national option 
in an international study that was 
planned for the third module of the 
Second International Technology in 
Education Study (SITES). (SITES was 
funded by the International Association 
for the Evaluation of Educational 
Achievement, or IEA.) The framework 
was intended to guide the develop- 
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Figure 1. Coordinated ICT assessment framework 


ment of performance assessments of 
ICT that could be used across a range 
of technology use in school subjects 
documented in IEA SITES Modules 1 
and 2 (Kozma, 2003). 

To these ends, a working group of 
international experts in ICT represent- 
ing Chile, Finland, Norway, Singapore 
and the United States was formed. 
The group aligned standards docu- 
ments that specified important tech- 
nology proficiencies with those that 
focused on mathematics and science 
(since NSF was the funding agency) 
and the role of technology within 
those domains. To create the coor- 
dinated ICT assessment framework, 
the descriptions and classifications of 
problem solving and inquiry from the 
maths and science frameworks were 
incorporated into the more general 
categories of information processing, 
knowledge management and com- 
munication categories in the technol- 
ogy proficiency frameworks. From 
these frameworks, the project team 
culled common categories of ICT use 


that could shape the coherent col- 
lection of evidence across studies of 
students’ abilities to use ICT in aca- 
demic domains. The cross-cutting 
framework laid out the knowledge 
and skills to be assessed. It served as 
the first component of an evidence- 
centered assessment design for ICT 
(Mislevy and Haertel, 2006). Figure 1 
presents a model of the coordinated 
ICT assessment framework. 

The circle depicts the subject matter 
domains — the content and processes 
of the disciplines of science and math- 
ematics addressed in the NSF project. 
Other academic domains in social 
science and the humanities were not 
included, although the generic frame- 
work could be applied to domains 
other than maths and science. The 
left side of the circle represents the 
declarative knowledge of the domain, 
which can vary from content-lean, 
factual knowledge to content-rich, 
schematic knowledge composed of 
interrelated concepts and principles 
(Baxter and Glaser, 1998). The right 
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side of the circle represents the proc- 
ess dimension, in which problem-solv- 
ing demands of an assessment can 
range from simple, procedural knowl- 
edge for routine problems to complex, 
strategic knowledge for nonroutine 
problems. Within the problem space, 
learners use ICT strategies to inte- 
grate technologies into the problem- 
solving activities. The ICT strategies 
include: taking advantage of the capa- 
bilities of technologies to understand 
and plan how to approach a problem; 
accessing and organising information 
and relevant data; representing and 
transforming data and information; 
analysing and interpreting information 
and data; critically evaluating the rele- 
vance, credibility and appropriateness 
of information, data and conclusions; 
communicating ideas, findings and 
arguments; designing products within 
constraints; and collaborating to solve 
complex problems and manage infor- 
mation. These strategies align with 
current versions of 21 st century skills. 

The figure deliberately portrays these 
ICT strategies as non-linear and itera- 
tive. Thus, planning may be needed 
to find relevant digital information and 
data at the outset of a task and again, 
at a later stage of the task, to decide 
what to vary in the test of a model. 
Various technologies can support col- 
laboration throughout the problem- 
solving activities. 

Technology tools appear in the center 
of the problem space in a ‘tool kit’. 
Internet, productivity and specialised 
tools such as simulations or visuali- 
sations may be chosen to accomplish 
multiple ICT strategies. Factual and 
procedural knowledge required for 
operation of specific tools or classes 
of tools can vary according to the 
affordances of particular tools and 
the basic or more advanced features 


chosen or required. This framework 
was designed to focus on generaliz- 
able ICT strategies, rather than on 
discrete, often changing, features of 
technology tools. 

New literacies assessment 
designs 

The NSF project involved design of 
prototype performance assessments 
that the international study could use 
to test problem-based reasoning using 
technology. The project used a modu- 
lar design approach that aimed to: 

provide common, credible, techni- 
cally sound measures of standards 
related to technology use, reasoning 
with information and communica- 
tion outcomes addressed in a wide 
range of technology programs and 
classrooms; 

• apply and extend an assessment 
design framework with modular 
components that could provide 
'templates’ or task models for new 
or modified assessments address- 
ing similar outcome areas; 
provide preliminary evidence about 
the technical quality of the general 
design approach and function of the 
prototype assessments. 

The modular design was intended to 
support flexible reuse of component 
tasks. First, the modules could be 
based on an ICT strategy, technology 
tool, subject-matter of the problem or 
complexity level. Second, modules 
could be independent of each other so 
that they could be inserted or deleted 
without disrupting the flow of an inves- 
tigation or problem-based assessment 
task. Third, the modular approach would 
permit extraction of separate score 
reports for domain knowledge, strat- 
egies and/or technology use. These 
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Module 

Sample questions/tasks 

ICT strategy 

Strategy 

component 

Sample tools 

1 

Given data in text message 
of 4 years of hare and lynx 
population data, describe the 
problem. 

Given data for more years by 
collaborators, describe the 
problem. 

Plan 

strategies and 
procedures. 
Collaborate 
to solve 
problem. 

Analyse problem. 
Choose 

appropriate tools. 
Integrate others' 
data. 

Spreadsheet 

Table 

E-mail 

2 

Type in a search to find how hare 
and lynx populations are related. 
Look through these three sites. 
Take notes and cite sources. 

Copy and paste information. 

Pick which search might be 
better. 

Are these good search results? 
Send suggestions to collaborator. 

Access 

information 

and data. 

Organise 

information 

and data. 

Critically 

evaluate. 

Collaborate. 

Formulate a search 
query. 

Conduct search. 
Enter information 
in table or notes. 
Evaluate quality of 
search results. 
Contribute 
feedback. 

Web browser 
Table 

Search box 
Search results 
Web directory 
Web pages 
Table 
Word 
document 
E-mail 

3 

Enter the 25 years of population 
data into a spreadsheet. 

Create another way to look at the 
pattern. 

What is the relationship in 2003? 
What trends do you see? 

What do you predict will happen 
in 5 years? 

Represent 
and transform 
data and 
information. 
Analyse and 
interpret data. 

Display data in one 
format, convert to 
a different form. 
Record and read 
data. 

Identify and 
explain trends. 
Make predictions. 

Spreadsheet 

Table 

Graph 

4 

Run the model with given 
settings. 

What are the populations in 2002 
and 2005? 

What do you predict will happen 
In 2008? 

Increase the lynx population. 
What do you think will happen? 
Run the model. Explain 

Analyse data. 
Interpret data. 

Read graphs. 

Infer trends. 

Make predictions. 

Explain 

predictions. 

Modelling tool 

Word 

processor 

5 

Plan your recommendation and 
presentation. 

Compose your presentation 
using information and pictures 
from websites, data. 

Present argument. 

Plan 

argument. 
Communicate 
findings and 
supported 
argument. 

Specify position 
Identify relevant 
evidence. 

Present 

recommendation, 
relevant data 
and information 
in coherent 
argument. 

Web form 

Word 

processor 

Tables 

Graphs 

Graphics 

Presentation 

tool 

6 

Critique recommendation from 
another team (with inaccurate 
data) by explaining if you agree 
with the recommendation, the 
appropriateness of their data and 
information, their support for the 
recommendation. 

Critically 

evaluate 

arguments. 

Critique position, 
evidence, support 
explanation, 
organisation. 

E-mail 

Word 

processor 


Table 1: ICT assessment scenario: predator-prey 

Problem: Parks are being overrun by hares. The government should reintroduce lynx. 
Science and math content: Familiar or given. 
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affordances of the modular design for 
ICT performance tasks would permit 
custom design and adaptive assess- 
ment. The next section presents exam- 
ples of prototype ICT assessments of 
and with technology that use the modu- 
lar design approach. 

Prototype tasks for 
assessing new literacies 

Assessment oftechnology: This first 
prototype was designed to illustrate 
modules that could be developed for 
the planned SITES 2 Module 3 optional 
performance assessments. This proto- 
type, funded by the National Science 
Foundation, was an assessment of 
technology, aimed to test student pro- 
ficiencies using common Internet and 
productivity tools to solve problem- 
based tasks drawing upon familiar sci- 
ence and maths content. The proto- 
type was designed to be appropriate 
for administration to 13-year-olds. The 
predator-prey ICT assessment pre- 
sented a driving authentic problem: 
‘Should lynx be reintroduced into a 
Canadian park?’ This problem was an 
example of predator-prey problems 
that have been addressed in curricula 
ranging from the upper elementary to 
university levels. For this prototype, 
the science and mathematics required 
were well-taught, well-learned mate- 
rial. Table 1 outlines the sequence of 
tasks. 

Questions and tasks within modules in 
this prototype were designed to cap- 
ture student responses dynamically 
as students employed ICT strategies 
to accomplish a subtask by using vari- 
ous technologies. First, the problem 
is presented and hypothetical student 
team members from another school 
who will be virtual collaborative part- 
ners are introduced. 


Module 1 assesses ICT planning 
strategies through questions and 
tasks for analysing the problem by 
examining data on hare and lynx 
populations, while selecting from 
a set of technology tools. Module 

1 assesses collaborative planning 
through tasks and questions in which 
the student uses e-mail to examine 
hare and lynx population data sent 
by virtual team members. Evidence 
of skills in operating the technology 
tools is a by-product of students’ use 
of the tools in the problem-solving 
tasks. 

Assessment of strategies for using 
technology to access and organ- 
ise information is tested in Module 

2 in a series of tasks in which the 
student formulates a search query, 
gathers information and data from 
web pages and organises them in 
a table. Critical evaluation, tested 
throughout the modules, is assessed 
by questions on the credibility of 
information from a web report pro- 
duced by a fur trading company and 
by questions on the effectiveness of 
web search results. 

Module 3 assesses the ICT strate- 
gies for using technologies to rep- 
resent and transform information 
and data. Questions and tasks ask 
students to convert data sent in an 
e-mail text message by virtual col- 
laborators to data on a spreadsheet 
and then transform the data into a 
graph. 

Module 4 tests the ICT strategies 
for using technologies for analysis 
and interpretation of information 
and data. Questions and tasks ask 
students to read specified data pre- 
sented in tables and graphs and to 
interpret trends. 

Module 5 tests analysis and interpre- 
tation by using a modelling tool that 
displays the pattern of hare and lynx 
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populations. Students answer ques- 
tions about output of the model at 
specified years, predict trends, and 
manipulate population values in the 
model to test predictions. 

• In Module 6, uses of ICT strate- 
gies and technologies for planning 
a presentation and communicating 
findings and results are tested. 

Figures 2 to 9 illustrate the modules. 

The predator-prey modules were 
designed to permit flexibility in inter- 
national administrations in which 
the assessment would be a national 
option. The modules such as using a 
spreadsheet or modelling tool could 
be removed if students had not had 
experience with these tools, but the 
flow of the problem-solving task would 
not be disrupted. 


Assessments with technology: 
solving complex science and math- 
ematics problems using advanced 
learning tools: A second assessment 
design goal in the NSF ICT assessment 
project was to draw on the coordinated 
ICT assessment framework and the 
modular design approach to fashion 
prototype performance assessments 
for the secondary school level that 
would tap transformative uses of learn- 
ing with technology in advanced sci- 
ence and mathematics (e.g. visualisa- 
tions, modelling, specialised software). 
The prototypes addressed assessment 
targets for science concepts, ICT strat- 
egies and the use of technology tools. 
The prototypes were designed to serve 
as classroom-level models for teach- 
ers and evaluators to assess student 
learning at the secondary level in inno- 
vative technology-supported curricula 
in which students had the opportunity 


Too many hares! 

• Read the following passage. 

Park rangers in Canada have observed that 
too many hares in the parks are causing 
problems. They eat small plants that other 
animals depend on for food. Some park 
rangers suggest that the government bring in 
more lynx (which eat hares) and help reduce 
the population. 


©B 

The park service hired Dr. Kloss at the Arctic 
Research Institute to investigate. Dr. Kloss wants 
teams of students made up of students from 
different schools to help. You'll be working with 
two other students, Filo from York and Kari from 
Ottawa. 

For this project you will use technology to solve 
the problem should the government bring in more 
lynx? 
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Collect information, take notes 
and cite sources 


Data I information to collect: 

• When does the hare population start to go down? 

• What are reasons that can cause it to go down? 


» Canadian lynx 
» The Ivnx and hare 
» Predator-prey cycles 



Figure 3: Predator-prey ICT assessment — Web search task 


Access & Organize Information / Data 
Analyze & Interpret - infer trends & patterns 

Data from the last 4 years 

Here is the data for the number of hares in the parks over the last four years. Last 
year (2002) there were about 95,000 hares. The year before that (2001 ) there were 
about 80,000. In 2000, there were 25,000. And in 1999, there were only about 
1 ,000 hares. 

Organize the data and describe the population trend. 

Pick a tool to use: 


Word processor H Spreadsheet Presentation 


o 


Finished with task 


Figure 4: Predator-prey ICT assessment — Select a tool to organise information 
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Collaborate 


Represent & Transform Information / Data 

More data 


\E] 


I saw the data that you got from the Canadian park service. I did some 
more research and found a list that goes back 25 years and shows 
how many hares and lynx there were. I attached the information in a 
spreadsheet but I can't understand it. We need to figure out how to 
make sense of all this. Can you find a better way to analyze and 
display the data? 


Thanks, 

Kari 





Open attachment 


o 


Finished with task 


Figure 5: Predator-prey ICT assessment — Organise data 
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Figure 6: Predator-prey ICT assessment — Transform data from table to graph 
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Use a model to predict outcomes 



Figure 7: Predator-prey ICT assessment — Use a model 


Create a presentation 

Dr. Kloss would like you to prepare a short presentation (3 pages / slides) to the park service 
with your recommendations what to do about about the hare and lynx populations. 

Be sure your presentation is clearly organized and includes: 

• A statement of the problem 

• Your group's recommendation 

• Information, data and explanations to support your recommendation 


Presentation H Word processor 


o 


o 


Finished with task 


Figure 8: Predator-prey ICT assessment — Select a tool to develop presentation 
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to work with the types of technological 
tools used by professionals. 

One prototype was designed to test 
the ability of high school physics stu- 
dents to apply the laws of motion to 
solve an authentic problem (design- 
ing a motorway car crash barrier) with 
a widely used commercial modelling 
tool, ‘Interactive Physics’. The tar- 
geted knowledge and skills included: 

(a) physics concepts related to force, 
mass, acceleration and velocity; 

(b) ICT inquiry strategies for planning/ 
design, conduct of investigations (run- 
ning the simulation), analysis and inter- 
pretation (of acceleration and velocity 
graphs), evaluation of possible design 
solutions and communication of a 
recommendation; and (c) technology 
proficiencies related to using the mod- 
elling tool, graphing tool and presenta- 
tion tools. The task design consisted of 
a series of modules in which students 
planned their design, iteratively pre- 
dicted and tried out designs using the 
simulation, interpreted results, evalu- 
ated a proposed design and devel- 
oped a presentation for their recom- 
mended design. Evidence of student 
learning was provided by scores for 
student work related to physics knowl- 
edge, the component inquiry skills and 


technology use. Figure 9 presents a 
screen shot of the module. 

Another prototype designed according 
to the modular design approach tested 
a student model for secondary stu- 
dents to solve an applied problem by 
using a widely available commercial 
visualisation tool, ArcView. The tar- 
geted knowledge and skills included: 

(a) science and maths knowledge; 

(b) inquiry skills for planning and con- 
ducting investigations, analysing and 
interpreting data and communicating 
recommendations; and (c) technology 
use. The task design involved pres- 
entation of the problem (Which states 
meet requirements to apply for solar 
power funds?); accessing, analysing 
and combining visualisations of dif- 
ferent types of data (for solar energy); 
interpreting data; and presenting a 
recommendation. Evidence of student 
learning consisted of scores for the 
three outcome areas and their com- 
ponents. Figure 10 presents a screen 
shot of the assessment. 

The three prototypes developed in the 
NSF project described above used the 
coordinated framework for the design 
of ICT assessments to illustrate how a 
modular design approach can shape 
assessments of new literacies that can 
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Figure 9: High school physics assessment using simulation software 
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ICT SOLAR POWER PERFORMANCE ASSESSMENT 

Procedural Knowledge - Operate tools 

Access & Organize - Search multiple representational formats 
Analyze & Interpret - Use visualization tools to investigate, 
compare 


Averuqe Sunshine • Average (Percent) 

Hours ot sunshine ms j pticontj j* of potential hours (1930- 1900 ) 



Figure 10: High school science and math assessment using visualisations 


vary foregrounding of competencies in 
use of the technology tools, in use of 
21st century ICT cognitive strategies, 
or domain knowledge and skills. The 
next section examines contemporary 
designs of technology-based assess- 
ments and their potential for providing 
evidence of student learning of new 
literacies advocated in 21st century 
and ICT skill frameworks. 

Towards the next 
generation of assessments 
of new literacies 

A new generation of technology-ena- 
bled assessments is transforming how 
testing is done and what gets tested 
(Quellmalz and Pellegrino, 2009). 
An increasing number of large-scale 
tests are embracing testing by tech- 


nology. These testing programs are 
capitalising on the capacities of tech- 
nology to support logistical assess- 
ment functions including test devel- 
opment, delivery, adaptation, scoring 
and reporting. A new generation of 
assessments, however, is attempting 
to move beyond logistical supports of 
testing by technology to reformulating 
task and item formats to test 21 st cen- 
tury thinking and reasoning processes 
with technology in order to overcome 
many of the limitations of conventional 
testing practices. 

In 2006, the Programme for 
International Student Assessment 
(PISA) conducted a pilot of computer- 
based assessment in science which 
used animations and simulations of 
phenomena such as energy flow in a 
nuclear reactor to test science skills 
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that could not be tested in the paper- 
based booklets. In 2009, PISAincluded 
electronic texts to test reading. Since 
2005, the US state of Minnesota has 
administered computer-based state 
science tests in grades 5, 8 and 11. 
These science tasks present anima- 
tions and simulations of laboratory 
experiments and phenomena such as 
the water cycle. In the USA, in 2011, 
the national assessment of educa- 
tional progress (NAEP) for writing, 
word processing and editing tools will 
be used in the computer-administered 
test for grade 8 and grade 12 students 
to compose essays. 

The large-scale tests described above 
are assessments of subject matter 
knowledge and processing skills, i.e. 
assessments of learning with tech- 
nology. Data is not collected on how 
well technologies are used, nor of the 
number of 21st century skills used, 
such as collaboration or multimedia 
presentations. In fact, these subject 
area tests are designed to minimize 
the requirements for knowing how to 
operate particular technology tools. 

Large-scale assessments of the new 
technological literacies that directly 
test and report on the spectrum of 21st 
century ICT skills are not yet availa- 
ble. A 2003 ICT feasibility test by PISA 
was conducted with a small sample of 
students in Japan, Australia and the 
USA. The study pilot was an assess- 
ment oftechnology which tested a set 
of ICT skills for access, management, 
integration and evaluation. Modules 
included uses of web (select relevant 
reliable site, search), desktop (email, 
database) and e-learning (science 
simulation) environments. Scored ICT 
proficiencies related to students’ abili- 
ties to correctly use the technologies. 
A full-scale ICT assessment was not 
funded by PISA. 


Recommendations for21 st century ICT 
assessments are turning from a pri- 
mary emphasis on summative goals to 
methods for assessing new literacies 
within school curricula. Assessment 
designs are seeking to harness tech- 
nology to measure understanding 
of complex and dynamic phenom- 
ena that were previously difficult to 
assess by conventional means. In the 
domains of reading and written com- 
position, ICT tools such as web brows- 
ers, word processors, editing, drawing 
and multimedia programs can support 
reading and writing processes. These 
same tools can expand the cognitive 
skills that can be assessed, including 
accessing and finding relevant infor- 
mation, integrating multiple sources of 
information, planning, drafting, com- 
position and revision. 

These assessments of learning with 
technology can vary along a continuum 
from static to animated and dynamic 
displays of information, data and phe- 
nomena and from static to interactive 
ways for students to solve problems 
and enter responses (Koomen, 2006). 
At the beginning of the continuum 
would fall technology-based assess- 
ments by technology intended to repli- 
cate paper counterparts. Assessments 
that would fall at a midpoint on the 
continuum may permit students to con- 
struct tables and graphs or they may 
present animations of science experi- 
ments or phenomena, such as chemi- 
cal reactions, for students to observe. 
Assessments presenting dynamic 
simulations that allow students to 
interact by manipulating multiple vari- 
ables would be placed at the most 
transformative end of the continuum. 
Technology-enhanced assessments 
can offer the following benefits. 

Present authentic, rich, dynamic 

environments. 
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Support access to collections of 
information sources and expertise. 
Present phenomena difficult or 
impossible to observe and manipu- 
late in classrooms. 

Represent temporal, causal, 
dynamic relationships ‘in action’. 
Allow multiple representations of 
stimuli and their simultaneous inter- 
actions (e.g., data generated during 
a process). 

Allow overlays of representations, 
symbols. 

Allow student manipulations/investi- 
gations, multiple trials. 

Allow student control of pacing, 
replay, reiterate. 

Make student thinking and reason- 
ing processes visible. 

Capture student responses during 
research, design, problem solving. 
Allow use orsimulations of a range of 
tools (Internet, productivity, domain- 
based). 

Across the disciplines, technologies 
have expanded the phenomena that 
can be investigated, the nature of argu- 
mentation and the use of evidence. 
The area of science assessment is per- 
haps leading the way in exploring the 
presentation and interpretation of com- 
plex, multi-faceted problem types and 
assessment approaches. Technologies 
are being used to represent domains, 
systems, models and data, and their 
manipulation, in ways that previously 
were not possible. Dynamic models 
of ecosystems or molecular structures 
help scientists visualise and communi- 
cate complex interactions. This move 
from static to dynamic models has 
changed the nature of inquiry among 
professionals and the way that aca- 
demic disciplines can be taught and 
tested. Moreover, the computer’s abil- 
ity to capture student inputs permits 
collecting evidence of processes such 
as problem-solving sequences and 


strategy use as reflected by informa- 
tion selected, numbers of attempts and 
time allocation. Such work involves 
reconceptualizing assessment design 
and use and tying assessment more 
directly to the processes and contexts 
of learning and instruction. 

Assessments of new literacies at 
the classroom level: The systematic, 
direct assessment of new literacies 
in classrooms remains rare. Although 
students may be taughtto use common 
and advanced tools, teachers tend not 
to have specific technological literacy 
standards to meet nor testing methods 
to gather evidence of student skill in 
using the technologies. Teachers are 
typically left on their own to figure out 
how to integrate technology into their 
curricula. The state of practice for 
assessing new literacies integrated 
into instructional activities remains in 
its infancy. 

The advent of the 2012 NAEP 
Tecnological Literacy probe will provide 
as set of examples of new literacies 
in areas of Technology and Society, 
Engineering Design and Systems, 
and ICT. In the USA, assessments of 
21st century skills and technological 
literacy standards are required for all 
students by grade 8; however, states 
may report achievement on a state 
test or from school reports. School 
reports may be based on teacher 
reports that may, in turn, be based 
on questionnaires or rubrics judging 
students’ use of ICT in project work. 
Most teachers do not have access to 
classroom assessments of 21st cen- 
tury skills or professional development 
opportunities to construct their own. 
Moreover, the lack of technical qual- 
ity of teacher-made and commercially 
developed classroom assessments is 
well documented (Wilson and Sloan, 
2000). Even more of a problem is the 


136 


New technological literacies 


lack of clarity for teachers on how to 
monitor student progression on the 
development of 21st century skills, not 
only tool use, but ways to think and 
reason with the tools. Teachers need 
formative assessment tools for these 
purposes. 

The UK ICT Stage 3 assessment pro- 
gramme represented an attempt to 
provide teachers with assessments 
to check and monitor their students' 
operation of ICT tools (National 
Assessment Agency, 2008). In a 2007 
pilot of an ICT test, modules on use of 
websites, databases, graphs, images 
and presentations were administered 
and teachers received feedback on 
where students’ proficiencies fell on 
a continuum of operational tasks. 
Teachers were then expected to help 
their students become more proficient 
with the ICT tools. A major challenge 
reported from the 2007 pilot was that 
teachers viewed the time required to 
prepare students to take the exams 
as time taken away from their regular 
instruction. This finding supports the 
need for assessments of 21st century 
ICT strategies and operations that are 
designed as assessments of learning 
with technology. 

For direct assessments of new lit- 
eracies knowledge and strategies to 
become integrated into classroom 
formative assessment practices, new 
literacies assessments must be sys- 
tematically designed and subjected to 
technical quality screening. The form- 
ative use of assessment has been 
repeatedly shown to significantly ben- 
efit student achievement (Black and 
Wiliam, 1998). Such effects depend 
on several classroom practice factors, 
including alignment of assessments 
with standards and frameworks, qual- 
ity of the feedback provided to stu- 
dents, involvement of students in self- 


reflection and action, and teachers 
actually making adjustments to their 
instruction based on the assessment 
results ('). Technologies are well- 
suited to supporting many of the data 
collection, complex analysis and indi- 
vidualised feedback and scaffolding 
features needed for the formative use 
of assessment ( 2 ). However, for the 
most part, technology-based assess- 
ments that provide students and 
teachers with feedback on perform- 
ance on the subject matter tasks and 
items do not also provide feedback on 
students’ use of embedded technol- 
ogy tools such as graphs, tables or 
visualisations. 

The next section describes assess- 
ments being developed by WestEd 
in a SimScientists project funded 
by the National Science Foundation 
(Quellmalz, Timms and Buckley, 
2009). The project is studying the use 
of science simulations for end-of-unit, 
summative, benchmark purposes and 
for curriculum embedded formative 
purposes. The project assesses com- 
plex science learning with technology. 
Students use a range of technology 
tools and inquiry skills to investigate 
science problems that relate to under- 
standing increasingly complex levels 
of grade-appropriate models of sci- 
ence systems. Assessment targets 
are integrated knowledge about a sci- 
ence system and inquiry skills aligned 
with 21st century skills such as analy- 
sis, evaluation and communication. 
Although the project does not directly 
assess students' use of technology 
tools or their abilities to select appro- 
priate tools for a task, this paper offers 


(') Black, P., Harrison, C., Lee, C., Marshall, B., 
and William, D. (2004). Phi Delta Kappan 86, 8. 
( 2 ) Brown, J., Hinze, S., and Pellegrino, 
J.W. (2008). In: 21st century education , 

T. Good (ed.), Sage, Thousand Oaks. CA, Vol. 2, 
Chap. 77, 245-255. 
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suggestions for how such assess- 
ments could be augmented with 
tasks, items and feedback to promote 
21st century ICT strategies such as 
tool selection and use or collaborative 
research. By providing formative feed- 
back and further scaffolding on the 
use of technologies as they are used 
during subject matter problem solving, 
the assessments can encompass new 
literacies and lessen teachers' percep- 
tions that technological fluency poses 
additional, irrelevant burdens. 

Figure 10 presents a screen shot 
of tasks in a SimScientists assess- 
ment designed to provide evidence 
of middle school students’ under- 
standing of ecosystems and inquiry 
practices. Students are presented 
with the overarching problem of pre- 
paring a presentation and report to 
describe the ecology of a lake for an 
interpretive centre. They investigate 
the roles and relationships of the fish 


and algae by observing animations of 
the interactions between and among 
organisms in the lake. The assess- 
ments then present sets of simulation- 
based tasks and items that focus on 
students’ understanding of the emer- 
gent behaviours of the dynamic eco- 
system by conducting investigations 
with the simulation to predict, observe 
and explain what happens to popula- 
tion levels when numbers of particular 
organisms are varied. In a culminating 
task, students write a report of their 
findings about the lake ecosystem. 

In a companion set of curriculum 
embedded assessments, the techno- 
logical infrastructure identifies types 
of errors and follows up with feed- 
back and graduated coaching. In the 
assessment screen shown, feedback 
is provided if the student’s investiga- 
tions saved do not show organisms 
existing for the specified amount of 
time. Levels of feedback and coaching 



Figure 11: SimScientists assessment screenshot — Using a model to conduct investigations about population 
dynamics 
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progress from identifying that an error 
has occurred and asking the student to 
try again, to showing results of inves- 
tigations that met the specifications. In 
the task shown, additional evidence 
could be collected on technological 
literacy. The system could score how 
well students are able to vary values 
for the number of organisms while 
using the simulation, use the graph 
inspector to examine the graphs and 
tables, and save and enlarge views of 
graphs of multiple experiments. Such 
additions would allow assessment of 
technology, i.e. students’ understand- 
ing of how and when to use the tech- 
nology features in the simulations, as 
well as assessment of learning out- 
comes with technology. 

Developing new literacies 
assessments with technical 
quality 

The report, Knowing what students 
know (Pellegrino et al. , 2001), sum- 
marises the implications for assess- 
ment of decades of research in the sci- 
ences of cognition and psychometrics. 
The report characterises the assess- 
ment of the knowledge and skills that 
individuals possess as a process of 
reasoning from evidence. The reason- 
ing required to make inferences about 
an individual’s knowledge and skills is 
best developed through the specifica- 
tion of an assessment argument that 
connects three components including: 
(a) the specific knowledge and skill 
constructs in the particular domain(s) 
to be measured; (b) the features of 
assessment activities that will require 
examinees to use that knowledge and 
those skills; and (c) the data derived 
from student responses that will count 
as evidence of the level of knowledge 
and skills demonstrated. These three 
components of evidence-centered 


design represent a best practice in 
the field of assessment (Mislevy and 
Haertel, 2008). Assessments of new 
literacies, then, should specify an 
assessment framework with these 
three components. The framework 
should identify the 21st century ICT 
domain knowledge and processes 
that define the constructs to be meas- 
ured. These would include students’ 
declarative knowledge about technol- 
ogy tools, such as their purposes and 
features, and students’ procedural 
knowledge, or proficiency for operat- 
ing particular technology tools. The 
21st century ICT domain would also 
define strategies such as information 
processing, knowledge management, 
problem solving and communication; 
each of these are strategies that indi- 
viduals must draw on to make use of 
technologies to address significant, 
recurring problems in general, applied 
contexts and in academic disciplines. 
For new literacy assessments aiming 
to measure technology use and also 
to measure academic knowledge and 
skills, the framework would need to 
specify, test and report separately 
21st century thinking and reasoning 
strategies including collaboration and 
communication, use of desk top or 
e-learning tools and domain knowl- 
edge and processes. 

The second component of evidence- 
centred design for the assessment 
of new literacies would then specify 
the features of assessment tasks and 
items that would elicit observations of 
achievement of the 21st century ICT 
and domain knowledge and skills of 
interest. The types of assessment tasks 
and items would represent the types of 
fundamental contexts, problems and 
activities in which examinees use tech- 
nology in school and applied settings. 
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The third component of evidence-cen- 
tred design would specify: (a) the evi- 
dence of student learning that needs to 
be extracted from student responses 
to the assessment tasks and items; 
(b) how the responses will be scored; 
and (c) the details of the statistical 
models needed to calibrate items and 
create proficiency estimates and reports 
of students’ knowledge and skills. 

By shaping large-scale and classroom 
assessments according to this princi- 
pled assessment design approach, new 
literacies assessments can initiate the 
process of documenting technical qual- 
ity by describing a systematic design 
process. Further technical quality evi- 
dence gathered during cognitive labs 
of students thinking aloud as they solve 
assessment tasks and items would 
provide evidence of construct validity. 
The psychometric data from analyses 
of student performance on tasks and 
items would provide further evidence of 
technical quality. Since the process of 
documenting technical quality requires 
considerable expertise, some of the 
assessment resources made available 
to teachers for classroom formative 
assessment should have such proc- 
esses and data documented. 

Multilevel, balanced 
assessment systems 

Discussions of the need for assess- 
ments of 21st century ICT competen- 
cies increasingly recognise the need 
for the articulation of large-scale and 
classroom-based assessments. A key 
feature in creating a multi-level, bal- 
anced system is the use of common 
design specifications that can operate 
across classroom, district, state and 
national levels (Quellmalz and Moody, 
2004). Deliberately designing assess- 
ments at different levels to assure their 


coherence will increase the validity of 
inferences from the assessments and 
increase the likelihood that informa- 
tion about student performance can be 
used to describe and promote skilled 
use of technologies in significant aca- 
demic and applied tasks. 

Summary 

The development of assessments of 
new literacies is in its early stages. 
Multiple frameworks, contexts and 
points of view both invigorate and com- 
plicate design efforts. Educators differ 
as to whether or not technology should 
be assessed as a distinct domain or 
should be integrated into assessments 
within academic disciplines (Quellmalz 
and Kozma, 2003). Expert panels need 
to reach consensus on the knowledge 
and skills that constitute new literacy 
skills and how those skills align with the 
knowledge and skills in subject matter 
frameworks and standards. Research 
is needed on how to design tasks that 
integrate the use of technologies into 
subject matter tests and how to directly 
test, extract and report the skill with 
which technologies are operated and 
strategically used. Experts need to 
identify the features and functions of 
technologies that are relevant to aca- 
demic and 21st century constructs of 
interest as well as those features that 
need to be controlled because they 
interfere with performance on tar- 
geted knowledge and skills. Studies 
are needed to examine student per- 
formance on items and tasks in which 
technology is assumed to enhance or 
hinder performance. 

Work with technology-based assess- 
ments that scaffold learning and 
performance in complex tasks while 
adapting to student responses is also 
in its early stages. Research on ways 
that these adaptive modules can serve 
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as formative and summative assess- 
ments is greatly needed. Changes 
in scaffolding could be features that 
are varied in the assessment tasks. 
Research would examine how changes 
in the scaffolding levels of assessment 
task designs relate to student perform- 
ance. Such efforts would provide the 
field with interdisciplinary 21st century 


ICT assessment frameworks, princi- 
pled assessment designs, exemplary 
assessments and evidence of their 
validity. In the 21st century students 
will need to become facile users of 
technologies, and 21st century edu- 
cators will need to be able to define, 
target, measure and promote students’ 
progress on these new literacies. 
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1. Introduction 

A major theme running through edu- 
cation policy recommendations (Catts 
and Lau, 2008; CERI, 2001; European 
Council, 2000; OECD, 2005; Unesco, 
2008) and policy initiatives (US 
Department of Education, 1996; CDC, 
2001; Singapore MOE, 1997, 1998) in 
many parts of the world is the impor- 
tance for education to prepare its citi- 
zenry for life in the 21st century. This 
has brought about changes in the 
school curriculum as well as plans for 
the integration of IT (') in the teach- 
ing and learning process to foster the 
development of 21st century skills in 
students. Is there evidence that these 
education policy initiatives impact on 
how teaching and learning take place 
in schools, and even more importantly, 
on students’ learning outcomes? In 
this paper, we explore this question 
in the context of the policy initiatives 
that have taken place in Hong Kong 
since 1 998, when the first ‘IT in educa- 
tion' masterplan was launched (EMB, 
1998), drawing on the data that have 
been collected over the period 1998 
to 2006 from international and local 
evaluation studies, with a particular 
focus on an evaluation study of stu- 
dents’ information literacy skills con- 


(') IT and ICT are used interchangeably to refer 
to information and communication technology. 


ducted as part of the evaluation of the 
effectiveness of the implementation of 
the first and second ICT in education 
masterplans (Law et al., 2007). 

This paper begins with an overview 
of the three ‘IT in education’ strate- 
gies (EMB, 1998, 2004; EDB, 2008) 
launched in Hong Kong to highlight 
the policy foci and the changes in em- 
phases that have taken place over 
time. It then summarises the changes in 
teaching practice and ICT use in Hong 
Kong schools between 1998 and 2006 
based on findings from international 
comparative studies of ICT in educa- 
tion. The design and key results from 
the evaluation study of students’ infor- 
mation literacy skills is then described. 
The paper ends with a discussion of the 
links between education policy, teach- 
ing practice and students' outcomes as 
revealed by the findings. 

2. ICT in education policies 
in Hong Kong since 1998 

The Hong Kong Government announ- 
ced its first ICT in education policy in 
November 1998 with its ‘Information 
technology for learning in a new era: 
Five-year strategy’ (EMB, 1998), as 
an integral part of the policy goal for 
Hong Kong to become ‘a leader, not 
a follower, in the information world 
of tomorrow’, which was a statement 
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made in the inaugural policy address 
of Mr Tung Chee Hwa, the first Chief 
Executive after the return of Hong 
Kong’s sovereignty to China in 1997 
(EMB, 1 998; p. i). The vision ofthis first 
policy was to help students develop an 
understanding of the pervasive impact 
of ICT on their daily lives and society as 
a whole, as well as higher order think- 
ing skills and abilities to seek, evalu- 
ate, organise and present information. 
The document indicates the need for 
schools to undergo a paradigm shift 
for the policy to be implemented suc- 
cessfully, though it does not elaborate 
on the nature of the shift. It highlights 
four important missions in order to 
achieve this vision. 

1. Access and connectivity — to pro- 
vide students and teachers with 
adequate and equitable access to 
IT facilities and access to informa- 
tion worldwide. 

2. Teacher enablement — to assist 
teachers’ migration to the new 
teaching mode. 

3. Curriculum and resource support — 
to meet the target of having 25 % of 
the school curriculum taught with 
the support of IT. 

4. Fostering a community-wide cul- 
ture — to coordinate all stakehold- 
ers within and outside the school 
sector (school management, teach- 
ers, students, parents, the busi- 
ness sector and other community 
bodies) to take up their new roles 
in ‘IT in education’ in a collabora- 
tive manner in implementing the 
policy. 

It is important to note that it was only 
in 2000, two years after the launch 
of the first five-year strategy, that the 
comprehensive curriculum reform ini- 
tiative to renew the school curriculum 
with the goal of preparing the younger 
generation for meeting the challenges 


of a knowledge-based society was 
launched (EC, 2000). This curriculum 
reform had a major impact on the for- 
mulation ofthe second ‘IT in education’ 
policy — Empowering Learning and 
Teaching with Information Technology 
(EMB, 2004). This document formu- 
lated the goal ‘to transform school 
education from a largely teacher-cen- 
tred approach to a more interactive 
and learner-centred approach’ (EDB, 
2004, p. i) as the ‘paradigm shift’ 
targeted. 

The vision of this second policy was 
to encourage the effective use of ICT 
as a tool for enhancing learning and 
teaching to ‘prepare the younger gen- 
eration forthe information age, turning 
schools into dynamic and interactive 
learning institutions, and fostering col- 
laboration among schools, parents and 
the community’ (EDB, 2004, p. 10). 
The document used a somewhat dif- 
ferent rhetorical language. Instead 
of missions, this document identified 
seven strategic goals: 

1 . empowering learners with ICT; 

2. empowering teachers with ICT; 

3. enhancing school leadership for 
the knowledge age; 

4. enriching digital resources for 
learning; 

5. improving ICT infrastructure and 
pioneering pedagogy using ICT; 

6. providing continuous research and 
development; 

7. promoting community-wide support 
and community building. 

These seven goals have a much 
stronger educational focus and reflect 
different priorities and a more compre- 
hensive set of strategies compared 
with the missions contained in the first 
policy. Empowering learning is identi- 
fied as the policy goal while the other 
six are strategic goals. There is an 
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underpinning assumption in this docu- 
ment that the process of IT implemen- 
tation involves innovation, the nature 
of which is not only technological, 
but also pedagogical. It is within this 
framework that enhancing school 
leadership — such that principals 
and key personnel in schools under- 
stand better the nature and process 
of change required — and continual 
research and development were given 
important strategic considerations in 
this second policy. 

The second policy was planned to pro- 
vide strategic guidance for three years 
in view of the fluidity in the technology 
and education arenas. The third ‘IT in 
education’ policy document — ‘Right 
technology at the right time for the right 
task’ — was released in 2008 (EDB, 
2008). As indicated by the title, IT is 
perceived as purely instrumental in this 
document; it does not see the need to 
identify what is ‘right’; and the focus is 
at the ‘task’ level rather than at the level 
of an overarching curriculum/educa- 
tional goal. Instead of identifying mis- 
sions (as in the first policy) or goals (as 
in the second policy), this third policy 
identified six strategic actions. 

1. Provide a depository of curricu- 
lum-based teaching modules with 
appropriate digital resources. 

2. Continue to sharpen teachers' ICT 
pedagogical skills. 

3. Assist schools in drawing up and 
implementing school-based ICT in 
education development plans. 

4. Enable schools to maintain effec- 
tive ICT facilities. 

5. Strengthen technical support to 
schools and teachers. 

6. Collaborate with non-governmen- 
tal organisations to raise informa- 
tion literacy of parents and launch 
parental guidance programmes on 
e-learning at home. 


This third policy is clearly a turnaround 
in the developmental direction taken 
by the first two. There is avoidance 
of any indication that there are value 
judgments in deciding how and what 
technology is used and that the vision 
and leadership of the school matters. 
It is a policy document in name with- 
out having to play the role of a policy 
without having to set a policy directive, 
with the least possibility of stimulating 
any debate or controversy. This policy 
was also released with an extremely 
low profile. There was no formal 
launch and no media publicity. It is not 
possible to pinpoint what might have 
caused such change and discontinuity 
in policy. However, there was a major 
change in the top-level leadership in 
the Education Bureau at the time this 
policy was drafted and approved, and 
the key people who led the curriculum 
reform launched in 2000 had stepped 
down. 

3. Teaching practice and 
ICT use in Hong Kong 
schools (1998 to 2006) 

Hong Kong took part in all three 
modules of the Second Information 
Technology in Education Study 
(SITES) conducted under the aus- 
pices of the International Association 
for the Evaluation of Educational 
Achievement (IEA). The first module, 
SITES-M1 (study homepage at http:// 
www.mscp.edte.utwente.nl/sitesm1), 
focused on describing the status of 
ICT and its use in schools through a 
survey of principals and technology 
coordinators, with data collection con- 
ducted at the end of 1998. Details of 
the design and findings from this study 
are reported in Pelgrum and Anderson 
(1999). This study collected informa- 
tion on the percentage of schools 
having ICT available for use for 
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instructional purposes within formal or 
informal educational settings, as well 
as the extent to which principals per- 
ceive ‘emergent’ practices in teach- 
ing and learning were present in their 
schools. Emergent practices were 
defined as those practices designed 
towards developing students' lifelong 
learning abilities. These are generally 
more student-centered, open-ended 
learning and teaching activities with 
characteristics not commonly found in 
traditional classrooms. These ‘emer- 
gent’ characteristics include the fol- 
lowing, and namely that students: 

develop abilities to undertake inde- 
pendent learning; 

learn to search for, process and 
present information; 
are largely responsible for control- 
ling their own learning progress; 
learn and/or work during lessons at 
their own pace; 

are involved in cooperative and/or 
project-based learning; 
determine for themselves when to 
take a test. 

The third SITES module, SITES 2006, 
was designed as a survey of schools 
and teachers to examine the kinds of 
pedagogical practices adopted in dif- 
ferent countries and the use of ICT in 
them. In this module, the principals 
were also asked the same question on 
their perception of the extent to which 
emergent practices were present in 
their schools. The mathematics teach- 
ers and science teachers surveyed in 
this study were also asked about the 
frequency with which different kinds of 
teaching and learning activities (tradi- 
tional as well as lifelong learning ori- 
ented ones) took place in their class- 
rooms and whether ICT was used in 
those activities. Details of the design 
and findings from the SITES 2006 


study are reported in Law, Pelgrum 
and Plomp (2006). 

As the first ‘IT in education’ strategy 
in Hong Kong was only launched in 
November 1998, computers were not 
used much for instructional purposes 
except for the teaching of computing- 
related subjects in the curriculum. 
Data collection for SITES-M1 was 
conducted at the end of 1998. The 
student-computer ratios in primary 
and secondary schools in Hong Kong 
were 53.3 and 35.7 respectively, 
which were rather low levels of hard- 
ware provisions among the participat- 
ing countries at the time (Pelgrum and 
Anderson, 1999). Use of computers for 
instructional purposes in non-comput- 
ing subjects was extremely rare. The 
SITES 2006 teacher survey results 
showed 70 % of mathematics teach- 
ers and 82 % of science teachers in 
Hong Kong reported having used ICT 
with the sampled grade 8 classes that 
they taught in that school year, which 
was among the highest percentage 
reported in the participating countries. 
This finding indicates that in terms of 
classroom adoption, the government 
strategies have achieved noticeable 
success. 

Obviously, use is not the only criterion 
for gauging success in policy imple- 
mentation. If ICT use were to support 
students’ development of 21 st century 
skills, it matters whether learning was 
still organised as traditional teacher- 
centered instruction or lifelong learn- 
ing oriented as characterised above, 
and how ICT was actually used in 
classroom settings. The SITES 2006 
teacher survey results indicate that 
the pedagogical orientation of Hong 
Kong teachers was among the most 
traditional among the 22 participat- 
ing educational systems (Law and 
Chow, 2008a). Further, the extent of 
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pedagogical deployment of ICT was 
slightly greater for traditional peda- 
gogical activities than lifelong learning 
oriented ones. On the other hand, the 
strong traditional pedagogical orienta- 
tion should not be interpreted as one 
of the policy outcomes. In fact, the 
SITES-M1 findings show that princi- 
pals in Asian countries, including Hong 
Kong, generally reported much lower 
levels of presence of emerging prac- 
tice in their schools compared with 
their counterparts in other participat- 
ing countries (Pelgrum and Anderson, 
1999). The results from the principal 
survey in SITES 2006 show a remark- 
able ‘swing’ in the percentage of prin- 
cipals reporting a lot of presence of 
emerging practice compared with the 
same statistic reported in 1998 — 
with countries like Norway, Slovenia 
and Denmark reporting big decreases 
while big increases were observed 
in Asian education systems such as 
Hong Kong, Japan, Thailand and 
Singapore, and in some others such 
as Israel and Italy. Findings from the 
two SITES studies seem to indicate 
that there has been a move towards 
more emerging, lifelong learning ori- 
ented pedagogical practices in Hong 
Kong classrooms over the period 1998 
to 2006, though practices as a whole 
are still very traditional because of the 
cultural and historical background of 
the schools. 

4. Assessing students' 
information literacy 
skills — Research design 

In 2006, the Centre for Information 
Technology in Education of the 
University of Hong Kong (CITE) was 
commissioned by the Education 
Bureau (EDB) to conduct a study 
on students’ information literacy (IL) 
skills as part of the overall evaluation 


of the second ‘IT in education’ policy 
(EMB, 2004). The goal was to find out 
whether students were able to make 
effective use of ICT to tackle learn- 
ing tasks in the school curriculum at 
a level that is not normally achievable 
without the appropriate use of tech- 
nology. In commissioning this project, 
the EDB was interested in methodo- 
logical innovation in assessment 
that will assess not only technical 
operational skills, but also students' 
problem-solving and lifelong learning 
skills. Hence the focus was less on the 
psychometric qualities of the evalua- 
tion indicators but more on exploring 
new ways of assessing new kinds of 
outcomes. In this section, we will elab- 
orate on the conceptual framework 
taken in this study with respect to IL, 
the key principles underpinning the 
design of the assessment tasks and 
a brief description of the assessment 
instruments, the technology platform 
used to conduct the assessment and 
the sampling design for the study. 

4.1. The conceptual framework 

In the curriculum reform document 
launched in 2000 (EC, 2000), IL is 
identified as one of the important skills 
for the 21st century. In this study, the 
conceptofILencompassesmuch more 
than simply technical competence 
and includes the cognitive abilities to 
identify and address various informa- 
tion needs, critically evaluate informa- 
tion and apply the learning gained in 
the solution of real-life open-ended 
problems. Furthermore, just as prob- 
lem solving requires not only generic 
‘problem-solving skills’ but also exper- 
tise in the relevant content knowledge 
as well as in the selection and use of 
tools appropriate to the problem con- 
text, IL is also subject-matter depend- 
ent. It is considered entirely possible 
that the level of IL achievement of a 
student may be different in different 
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Figure 1 : Conceptual framework of IL in the evaluation study 


subject domains. Hence, the assess- 
ment of IL should also take account 
of the domain context. Figure 1 is a 
diagrammatic representation of the 
conceptual framework underpinning 
this study on how IL develops in the 
context of learning within school cur- 
riculum subjects. 

In this framework, IL encompasses 
both cognitive and technical profi- 
ciency. Cognitive proficiency refers to 
the desired foundation skills of every- 


day life at school, at home and at work. 
Literacy, numeracy, problem-solving 
and spatial/visual literacy demonstrate 
these proficiencies. Technical profi- 
ciency refers to basic knowledge of 
hardware, software applications, net- 
works and elements of digital technol- 
ogy. These proficiencies are developed 
through acquiring generic technical IT 
skills and applying them for interac- 
tive learning within the corresponding 
subject learning contexts in everyday 
learning and teaching practices. 
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4.2. Identifying indicators 

for assessing students' 

information literacy 

In order to identify appropriate indi- 
cators for evaluating the impact of 
ICT in the learning of specific sub- 
ject disciplines, several major frame- 
works developed in different countries 
for the assessment of ICT literacy 
have been carefully reviewed (EMB, 
2005; ETS, 2003; MCEETYA, 2005). 
Somewhat different terminologies are 
used in these documents, but the set 
of skills these refer to are largely simi- 
lar. Further, these all adopt a process- 
driven approach in identifying indica- 
tors for IL. Two important features of 
the ETS framework make it most ame- 
nable for our adoption in the present 
study: its focus on IL as exhibited in 
complex tasks that resemble real-life 
situations, and the fact that it was 
designed for use in online assess- 
ment. Table 1 presents the details of 
the seven dimensions of IL compe- 
tence in the ETS framework. 

In implementing the IL framework for 
assessing students’ outcomes, we fur- 


ther developed a rubric with four levels 
of performance: novice, basic, profi- 
cient and advanced. Rubrics are scales 
of performance that can be used to 
judge the quality of students' perform- 
ance based on the descriptive criteria 
provided (Popham, 2003). Rubrics 
are considered appropriate for use in 
this study as they can be used across 
a broad range of subjects in assess- 
ing both the process and product of 
students’ learning (Moskal, 2000). 
Moreover, in assessing complex com- 
petences, rubrics providing specific 
objective criteria for different levels of 
performance offer a way to provide the 
desired validity in the grading process 
(Morrison and Ross, 1998; Wiggins, 
1998). The IL rubric developed in the 
present study is modified from NCREL 
(2003), which has been validated by 
Lee (2009) for use in assessing stu- 
dents’ I L outcomes as indicated by their 
performance in learning activities in the 
classroom and through their authentic 
learning products. Some examples of 
the rubrics and their application in the 
context of specific tasks in the PAs 
developed in this study will be given 
later in this paper. 


Define 

Using ICT tools to identify and appropriately represent information needs 

Access 

Collecting and/or retrieving information in digital environments 

Manage 

Using ICT tools to apply an existing organisational or classification scheme to in- 
formation 

Integrate 

Interpreting and representing information, such as by using ICT tools to syn- 
thesise, summarise, compare and contrast information from multiple sources 

Create 

Adapting, applying, designing or inventing information in ICT environments 

Communicate 

Communicating information properly in its context (audience and media) in 
ICT environments 

Evaluate 

Judging the degree to which the information satisfies the needs of the task 
in ICT environments, including determining authority, bias and timeliness of 
materials 


Table 1 : The seven dimensions of IL in the ETS framework adopted in this study (Source: ETS, 2003, p. 18) 
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4.3. The performance 

assessment tasks and their 
design considerations 

The evaluation study was to be con- 
ducted at two levels, grade 5 and 
grade 8, with three sets of online per- 
formance assessment (PA) tasks at 
each level: technical IL, mathemati- 
cal IL and Chinese language IL at 
grade 5 and technical IL, science IL 
and language IL at grade 8. As it was 
expected that students' technical IL 
competence might differ widely for 
students within the same educational 
level, it was decided that the techni- 
cal IL assessment tasks at both levels 
should be the same to allow for com- 
parison across the two age groups. 
Hence, a total of five sets of online 
performance assessment (PA) tasks, 
answer keys and scoring rubrics were 
developed for this study. 

Each set of PA was designed accord- 
ing to the following criteria. 

The contexts for the tasks within 
each PA are relevant to students’ 
daily life experiences and hence 
present authentic scenarios. 

With the exception of the technical PA, 
the PAs were designed to be relevant 
for and appropriate to the curriculum 
at the respective subject and grade 
levels. 

Each PA was designed to be com- 
pleted in 45 minutes. 

The full score for each PA was 50. 
The score for each question was 
approximately proportional to the 
time allocation for its completion. 
Each PA was designed such that 
the totality of tasks within the PA will 
provide assessment on all the seven 
IL dimensions. However, the levels 
of achievement required for satis- 
factory task completion may differ 
across the different IL dimensions. 


The number of tasks that assess 
achievement for each of the dimen- 
sions may also vary across the dif- 
ferent PAs, depending on the sub- 
ject disciplines with respect to their 
subject nature. 

For each PA, general guidelines 
will be given at the beginning of 
the assessment to the students for 
answering the questions. Besides, 
the approximate completion time for 
each main question is indicated at 
the end of the question in each PA. 

4.4. Example performance 
assessment items 
illustrating the IL dimension 
they assess 

Some examples of assessment items 
drawn from the technical PA, math- 
ematic PA and science PA are given 
below in this section to illustrate how 
the dimensions of IL are assessed in 
the different subject areas. 

4.5. Developing and using 
rubrics to assess students' 
performance 

As described in Section 4.2, we have 
developed for each IL dimension a 
generic set of assessment rubrics (i.e. 
descriptive criteria) for identifying per- 
formance at the four different levels: 
novice, basic, proficient and advanced. 
Based on these generic rubrics, a set 
of task-specific scoring rubrics was 
developed for each assessment item in 
each of the PAs. Table 2 presents the 
scoring rubric for item 3.1 in the science 
PA (see Figure 6). The item asked stu- 
dents to construct a classification dia- 
gram for a set of plants and animals. 
There are two IL dimensions involved 
for the satisfactory completion of this 
task: manage (apply an existing organ- 
isational or classification scheme for 
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An item to assess the 
‘define’ dimension. Figure 2 
shows an item in the 
technical PA designed to 
assess students’ ability 
to define their information 
needs. Here, students 
are asked to plan a trip 
for their grandfather and 
grandmother to visit Hong 
Kong. It asked the students 
to define appropriate 
keywords for searching 
the ‘discover Hong Kong’ 
website. The assessment 
criteria are related to 
whether students can 
identify the appropriate 
keywords or not. 


Figure 2: An item in the technical PA designed to assess the 'define' dimension 


Planning a trip 



You and your classmates are asked to form a group of 
three to do a project about ‘Planning a trip for your 
grandfather and grandmother 1 . It is the first time that 
your grandfather and grandmother visit Hong Kong and 
they would like to go to some special and distinctive 
scenic spots in Hong Kong. You and your groupmates 
are required to suggest 2 scenic spots in Hong Kong and 
make a presentation for the whole class to present why 
you have chosen these scenic spots. 

Question 1.2 

What are the keywords which you have used for you 
searching? 


A day trip to the Hong Kong 
Ocean Park 



Question 1 

John and his family would like to visit the Hong Kong 
Ocean Park this Saturday. Below is the information about 
his family. They do not have Smart Fun Annual Pass. 
Please go to the Internet, find out the ticket prices and 
help him to calculate the total amount for the whole 
family’s general admission for one day at the park. (10 
mins) 


An item to assess the 
‘access’ dimension. Figure 
3 shows an item in the 
mathematics PA designed 
to assess students’ ability 
to access information 
effectively, in this item, 
students are asked to use 
a search engine to retrieve 
correct fares for adults and 
children to visit the Hong 
Kong Ocean Park. The 
assessment criteria are 
related to whether students 
can access relevant and 
correct information or not. 
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Managing Information 



Question 2 

Your groupmate. Susan, also finds soma Information 
from tha Internet and puts them to a Word document. 
However, she is not familiar with the software and she 
would like you to make some changes for her document 
according to the following sample document (required 
formats have been highlighted), so that the information 
can be better organized. Apart from the above required 
changes. You may also use your own Ideas to make 
some changes for her document In order to make it 
more presentable. (10 mins) 



An item to assess the 
'manage' dimension. Figure 
4 shows an item in the 
technical PA designed to 
assess students’ ability 
to manage information 
effectively. This item 
asked the students to 
edit the information in a 
Word document according 
to six given formatting 
requirements. Students were 
also provided with a sample 
text formatted according 
to those six requirements 
for their reference. The 
assessment criteria are 
based on the number of 
changes that students can 
make correctly. 


Figure 4: An item in the technical PA designed to assess the manage' dimension 


Question 3 

Mary bought 144cm ribbon from the gift shop to decorate a 
rectangular photo frame. 


Use the mouse to drag the rectangle and observe the 
changes of the length and the width of the rectangle with 
perimeter 144 cm. (10 mins| 

OtMStfMl J.f 

Record Three sen of lengths, widths and areas that you 
have observed 



L»ngth (cm| 

Width |cm| 

Altai cm 1 ) 

Eg. 

4 

68 

272 

■ 




2 




1 






An item to assess the 
'integrate' dimension. Figure 
5 shows an item in the 
mathematics PA designed 
to assess students’ ability 
to integrate information 
effectively. In this item, 
students are asked to 
manipulate an interactive 
applet to observe changes in 
the area of a rectangle with 
the different length-width 
configurations formed 
by a piece of string of 
fixed length. Students 
are then asked to deduce 
the maximum area of 
a rectangle that can be 
enclosed by the piece of 
string. The assessment 
criteria are based on the 
comprehensiveness of the 
students' manipulations 
and observations, and the 
correctness of the students’ 
interpretations. 


Figure 5: An item in the mathematics PA designed to assess the 'integrate' dimension 
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Endangered Species 

Question 2 

Some new plants and animals have been sent to the 
Kadoorie Farm for adding to their collection. Your 
classmates used a digital camera to take pictures of 
these new species as shown below. 

Among the animals, there is an endangered species 
found in Hong Kong. An endangered species is a species 
whose number is so small that it is at the risk of 
extinction. Can you identify which species it is and what 
kind of habitat it needs? Use the links on the right hand 
side or other websites to look for information about 
endangered species in Hong Kong. (5 mins) 



Grantham's Camellia Bud Net Fern Golden Aijouti 


Question 3.1 

Please construct a classification diagram: 
i. Classify the above new animals and plants into four 
suitable categories. Please make reference to the 
"Nature Walk Self-guided Map” to help the farm staff 
decide where to place the above newcomers. 


An item to assess the 
‘create’ dimension. Figure 6 
shows an item in the science 
PA designed to assess 
students’ ability to effectively 
create representations of 
information. In this item, 
students were asked to 
use electronic resources to 
create a classification chart 
with four categories for nine 
species and also include 
both the names and photos 
of those species in the chart. 
The assessment criteria are 
based on the complexity of 
the chart created. 


Figure 6: An item in the science PA designed to assess the 'create' dimension 


An item to assess the 
‘communicate ’ dimension. 
Figure 7 shows an item in 
the technical PA designed 
to assess students’ ability 
to communicate information 
effectively. This item asks 
students to share and 
discuss their suggestions 
on their choice of scenic 
spots for their grandparents 
using a discussion forum. 
The assessment criteria 
are based on whether the 
students can post their ideas 
and give responses to their 
peers or not. 

Figure 7: An item in the technical PA designed to assess the 'communicate' dimension 


Discussion 

Question 4 

You would like to discuss about the scenic spots you 
suggested with your classmates through the discussion 
Forum. In the Forum, you may discuss with your 
classmates why you suggest these two scenic spots 
and give opinions to those scenic spots suggested by 
your classmates. (5 mins) 


Click here to start the discussion 
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An item to assess the 
tevaluate ’ dimension. 

Figure 8 shows an item in 
the science PA designed to 
assess students’ ability to 
communicate information 
effectively. In this item, 
students have to run and 
observe the behaviour of an 
ecological simulation, and 
then propose a guideline 
for protecting the pond 
ecosystem. Hence they 
need to be able to evaluate 
the challenges to the pond 
ecology based on their 
observations of the simulation 
as well as what they have 
learnt from other information 
sources they read. The 
assessment criteria are 
based on whether students’ 
generated guidelines applied 
to the whole ecosystem and 
wether sufficient reasons 
were given. 

Figure 8: An item in the science PA designed to assess the evaluate' dimension 


Question 4-7 

Part II Exploring Ecosystems 
Using a Simulation Programme 



Doing ecological experiments may bring danger to the 
environment. Therefore. scientists often use 
simulations to find out how changes will affect different 
species. Now you are going to use one simulation 
programme to find out more about a pond ecosystem ( 
There are totally four questions in the 
simulation programme. (20 mins) 

Please read the instruction carefully once the simulation 
is started. 

7.4 Use the ‘chat room* to discuss with your classmates and 
draw up for the visitors one guideline to protect Hie pond 
ecosystem from any destructive behaviour done by tire 
visitors. Give explanations for this guideline. 


the information) and create (adapting, 
applying, designing or inventing infor- 
mation in ICT environments). Hence 
two scoring rubrics are necessary for 
assessing these two aspects of the stu- 
dents’ performance. The scoring rubric 
shown in Table 2 is for scoring perform- 
ance in the create dimension only. The 
specific skill pertaining to the create 
dimension in this task is the ability to 
use an advanced tool to create a well- 
structured chart. The scoring criteria 
and an illustrative sample of students’ 
work for each level of performance are 
also provided in Table 2. 

Experienced teachers were recruited 
to score the students’ performance 
based on the students’ responses to 
the questions as well as the products 
they created for the assessment. The 


scoring of the PA tasks requires expert 
judgment based on a thorough under- 
standing of the scoring rubrics. A train- 
ing workshop including an inter-coder 
moderation and discussion of discrep- 
ant scoring was conducted before the 
formal scoring took place. The inter- 
coder reliabilities for the scoring were 
0.95 in mathematics. 0.99 in Chinese 
language at grade 5, 0.96 in Chinese 
language at grade 8, 0.95 in science 
and 0.98 in the technical PA for both 
grades 5 and 8. 

4.6. Challenges encountered in 
the design of performance 
assessment tasks 
in this study 

We encountered serious challenges 
in the design of the PA tasks. A corn- 
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prehensive literature review conducted 
at the start of the study revealed that 
most of the reported empirical work on 
assessment of IL was in the area of 
assessing technical IL (e.g. ETS, 2003; 
Lennon et al. 2003; Jacobs, 1999). 


Assessment of IL in subject-specific 
contexts was only found for science 
(e.g. Quellmalz et al., 1999; Quellmalz 
and Kozma, 2003). Hence the develop- 
ment of PA for technical IL and in differ- 
ent subject domains using the same IL 


IL dimension 

and specific Performance 
IL skill level 

assessed 


Scoring 

criteria 


Illustrative sample of students' work 
at this level 


Create - able 
to use an 
advanced 
tool to create 
a well- 
structured 
chart 


Advanced 


Proficient 


Basic 


Novice 


Able to use 
an advanced 
tool (diagram 
function, 
Excel or other 
drawing tool) 
to create a 
chart with 
at least 2 
levels of 
hierarchical 
structure 





Able to use 
an advanced 
tool (diagram 
function 
or other 
drawing tool) 
to create a 
chart with 
1 level of 
hierarchical 
structure 





Ooidan Aooixi 
Scwla'-baetad 

FlowacoKfcox 

uara o&turutrans 



Able to use 
a simple 
tool (table) 
to create a 
classification 
table 


Or 



Unable to 
create a 
classification 
chart 


n»mrvin»*i 

m 
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framework is a pioneering attempt we 
have made in this study. There are a 
number of challenges we faced in the 
design of the PA tasks that have not 
been satisfactorily resolved. Two of the 
challenges with important methodologi- 
cal implications are reported below. 

One of the methodological challenges 
in comparing students’ IL perform- 
ance across subject areas is the task 
dependence of the level of IL perform- 
ance required for the most satisfac- 
tory completion of a task along a spe- 
cific dimension. For example, an item 
assessing the ‘create’ dimension in the 
mathematics PA as shown in Figure 5 
(question 3.1) only asks students to use 
an interactive program to create three 
rectangles and record their lengths 
and widths. The level of performance 
required for the most satisfactory com- 
pletion of this task is at the basic level 
only. On the other hand, question 3.1 in 
the science PA (See Figure 6) assess- 
ing the same dimension (create) 
required students to create a classi- 
fication diagram. For the mathemat- 
ics item, it simply required students to 
follow the instruction to create an arti- 
fact. On the other hand, the science PA 
required a higher level of competence 
for satisfactory task completion since 
they need to determine the shape of 
the chart and how many hierarchical 
levels they need for the chart. In other 
words, the levels of skills and com- 
petences in the ‘create’ dimension in 
the science PA are higher than those 
required in the technical PA. Matching 
the levels of performance required for 
all dimensions on two different PAs is 
particularly difficult if the task contexts 
in both have to be authentic. 

Another challenge for ‘comparability’ 
across subject domains is that there 
are some digital tools and their usage 
which are core IL performance only for 


specific subjects because of the nature 
of the learning tasks and the discipline. 
Examples of these are exploratory 
geometry tools in mathematics and 
simulation tools for exploring the out- 
come of different scenarios in science. 
In the present study, in assessing ‘eval- 
uation’ skills in science, students were 
required to run the simulation program 
and make observations of how the 
ecology changes and then discuss with 
their peers to propose a guideline for 
protecting the pond ecosystem (see 
Figure 8 for task detail). However, for 
evaluating the same dimension in the 
technical PA, students only need to 
critically evaluate whether the retrieved 
information was related to the topic, 
without the need for using any sub- 
ject specific tool. It is also not possible 
to isolate the effect of the students’ 
subject matter knowledge on their IL 
performance. 

4.7. Sampling and 

administration of the 
assessment 

A two-stage sample design was used in 
this study. First, 60 primary schools and 
60 secondary schools were sampled 
from each of these two populations of 
schools using stratified random sampling 
based on the achievement banding ( 2 ) 
and size of the schools. Then one intact 
class (at grade 5 for primary schools 
and at grade 8 for secondary schools) 
is randomly selected from each sam- 


( 2 ) All students in publicly funded primary schools 
need to take territory-wide assessment for the 
purpose of secondary school placement. Primary 
schools can be categorised into three student 
achievement groups (referred to in Hong Kong 
as achievement banding), high, medium and low, 
based on the performance of their students in 
these assessments. Secondary schools can also 
be categorised into three achievement bands 
based on the assessment performance of the 
students they admit into secondary 1 . 
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pled school to take part in the IL assess- 
ment. As both the assessment of IL and 
online performance assessment were 
totally new to schools, it was not easy to 
get schools to voluntarily agree to take 
part in the study. At the end, 40 primary 
and 33 secondary schools took part in 
the study (after replacement). A total of 
1 320 grade 5 students and 1 302 grade 
8 students took part in the main data 
collection in this study. Two pilot studies 
had been conducted before the main 
data collection. The first pilot study was 
to ensure the validity of the instruments 
and the second pilot study was to try out 
the logistic arrangements of the main 
study. 

To ensure that the assessment 
conducted in this study was fair and 
valid, it was necessary for students in 
all schools to have access to a uniform 
computing environment. As a result of 
the implementation of the first ‘IT in 
education’ strategy, all schools in Hong 
Kong had been equipped with at least 
one computer laboratory with broad- 
band Internet access. However, the 
differences in hardware and software 
infrastructure and configurations were 
still extremely wide between schools 
in Hong Kong. It was also administra- 
tively not feasible for the study team to 
install the same software environment 
in the computer laboratories of the 
sampled schools. After exploring pos- 
sible alternatives, we decided on the 
use of a remote server system — the 
Microsoft Windows Terminal Server 
(WTS) — as the most suitable tech- 
nology platform for the administration 
of the IL performance assessments in 
our context. Students worked on the 
PA tasks in the computer laboratories 
in their own schools, which acted as 
dumb terminals. All assessment task- 
related computations and manipu- 
lations were in fact carried out and 
saved on the WTS. 


5. Students' information 
literacy outcomes — 
Impact of eight years of 
ICT in education policy 
in Hong Kong 

We analysed the results of the assess- 
ment by computing the percentage 
score obtained by the students for the 
items in each dimension. In the follow- 
ing section, we will first report on the 
technical IL achievement of the grade 
5 and grade 8 students to examine 
the differences between them. Next, 
a summary of the results across the 
three PA at the grade 8 level is pro- 
vided. A comparison of the students’ IL 
achievement across the three domains 
(technical, Chinese language and sci- 
ence) is then given, taking account of 
the challenges mentioned and of the 
limitations of the study, as pointed out 
in section 4.6. 

5.1. Students' performance in 
the technical IL PA 

Figures 9a and 9b shows the boxplots 
of the school means of students’ 
outcomes on the seven IL dimensions 
in the technical PA at grades 5 and 8 
respectively. These results show that 
at both levels performances in the 
dimensions of ‘define’, ‘access’ and 
‘manage’ were rather high while the 
poorest performances were observed 
in the dimensions of ‘communicate’ 
and ‘create’. The results also showed 
that grade 8 students had significantly 
better performance than grade 5 
students with respect to all seven IL 
indicators, which is not surprising. 

On the other hand, there were huge 
differences in students’ IL outcomes 
between schools at both levels. The 
largest dispersions were found in the 
dimensions of ‘define’ and ‘manage’. 
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The very large inter-school differences 
lead to some rather surprising obser- 
vations. Firstly, the mean achievement 
of the best-performing primary school 
was almost the same or higherthan the 
median school mean of the second- 
ary schools for all the IL dimensions 
with the exception of ‘manage’. On 
the other hand, the lowest-performing 
secondary schools had means below 
the median of all the school means 
for the primary schools except for the 
‘evaluate’ dimension. 

Results also showed that there were 
significantdifferences between schools 
in terms of students’ levels of IL com- 
petences in technical proficiency. This 
seems to indicate that school experi- 
ences matter in contributing to the IL 
outcomes of students and that the 
number of years of schooling and cog- 
nitive maturity contribute less to stu- 
dents’ IL outcomes compared with the 
curriculum experiences of students. 


5.2. Grade 8 students' 

performance in the Chinese 

language PA 

Figure 10 shows the boxplots of the 
school means of students' outcomes on 
the seven IL dimensions in the Chinese 
language PA for grade 8 students. It can 
be seen that performance was best for 
the dimensions ‘define’, ‘access’ and 
‘manage’ and significantly worse for the 
‘evaluate’ dimension. Further, compared 
with the performance in the technical PA, 
it can be seen that the medians of the 
school mean achievement levels across 
the different IL dimensions are much 
more similar (the ‘evaluate’ dimension 
being an outlier in this respect). 
On the other hand, the between- 
school differences in mean student 
achievement remain huge across 
schools, indicative of similarly huge 
differences across schools in terms of 
students’ opportunities to learn IL skills 
in the Chinese language subject. 



Figure 9a: Boxplots of the school means of grade 5 students' IL performance in the technical PA across the 40 
primary schools 



Teacher practices and student outcomes 



Figure 9b: Boxplots of the school means of grade 8 students' IL performance in the technical PA across the 33 
secondary schools 
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Figure 10: Boxplots of the school means of grade 8 students' IL performance in the Chinese language PA across 
the 33 secondary schools 
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Figure 11: Boxplots of the school means of grade 8 students' IL performance in the science PA across the 
33 secondary schools 


5.3. Grade 8 students' 

performance in the science PA 

Figure 11 shows the boxplots of the 
school means of students’ outcomes 
on the seven IL dimensions in the 
science PA for grade 8 students. It can 
be seen that performance is better in 
the ‘access’ and ‘define’ dimensions 
and poorest in the ‘evaluate’ dimension. 
Similar to the other assessment results 
reported earlier, the between-school 
differences in terms of the mean 
student achievement were very large. 
The variation in student performance 
across the seven dimensions as 
reflected by the medians of the school 
means differ much more widely than 
that found in the Chinese language 
PA but is somewhat smaller than that 
in the technical PA. However, when 
comparing students’ results among the 
five PAs, science has the lowest mean 
total score. 


5.4. Summary of the findings 
from the IL performance 

assessment study 

Overall, we find that most students have 
some basic technical skills in operating 
the computer, using the basic functions 
in the Office suite of applications and 
surfing the web. Student competence 
in lower-level IL skills such as defining 
and accessing information are highest 
while performance in the dimensions 
‘integrate’, ‘create’, ‘communicate’ and 
‘evaluate’ were poor. Student perform- 
ance was found to be poorest for items 
requiring the use of digital tools specific 
to the subject discipline, e.g. explora- 
tory geometry tools in mathematics 
and simulations in science. The inter- 
school differences in achievement also 
tend to be very wide for such items. We 
also find that schools with high achieve- 
ment banding do not necessarily have 
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higher overall student IL achievement 
levels. In fact, some newer schools 
with medium student achievement 
banding known for their engagement in 
curriculum and pedagogical innovation 
showed higher student IL achievement 
than some schools well-known for their 
excellent general academic achieve- 
ment. These findings indicate that IL 
achievement in the subject areas is not 
only dependent on students’ achieve- 
ment levels in the specific subject area 
assessed, but also on how ICT has 
been integrated into the curriculum by 
teachers in their classrooms. There 
was wide variation within schools and 
between schools in terms of student IL 
achievement, indicating that both stu- 
dent background and learning experi- 
ence in school matters. 

6. Conclusion 

What have the first two ‘IT in education' 
strategies in Hong Kong achieved? The 
studies reviewed in this paper indicate 
that some basic measures of infra- 
structure and teacher use have been 
achieved in all publicly funded schools 
in Hong Kong. There have been some 
changes in pedagogy, but pedagogi- 
cal innovation integrated with ICT use 
is still rare and not well integrated with 
use of ICT tools specific to subject 
areas. Students have generally gained 
some basic IT operational skills but are 
very poor in tackling the more complex 
tasks involving information literacy skills 
in 'integration', 'evaluation', ‘create’ and 
'communicate'. The findings also indi- 
cate that learning experience in school 
matters in terms of students’ I L achieve- 
ment and that there is still a long way 
ahead between students’ ICT use in 
classrooms and nurturing 21st century 
skills in Hong Kong. 

Analyses of the SITES 2006 data indi- 
cate that school leadership impacts 


on teachers' pedagogy (Law, 2008), 
which in turn also influences the per- 
ceived impact of ICT on students’ 
learning outcomes (Law and Chow, 
2008b). Further, in-depth analyses 
of the SITES 2006 and SITES-M1 
findings indicate that system level 
policy impacts on teachers' peda- 
gogical practice orientation and ICT 
use (Law, Lee and Chan, in press). 
The analysis of the policy changes in 
Hong Kong, both in terms of the first 
two ‘IT in education’ strategies and 
the overall school curriculum reform 
which started in 2000, has resulted in 
a stronger lifelong learning orienta- 
tion in pedagogical practices in Hong 
Kong classrooms. The various inter- 
national and local studies indicate 
that the policy initiatives have brought 
about positive (though yet still small) 
progress in realising the goal of lev- 
eraging the use of ICT to prepare stu- 
dents for life in the 21st century. The 
apparent change in policy direction in 
the third strategy is hence somewhat 
worrying. It has lost the strong focus 
on pedagogy and fostering of school 
leadership for ICT use in schools to 
support curriculum innovation, which 
have been found to be most important 
for achieving the educational poten- 
tial of ICT. Another concern is the 
absence of any mention of research 
and development as a strategic goal 
in the third strategy. The continuing 
support for local ICT-related research 
initiatives as well as Hong Kong's 
participation in the SITES studies 
have provided valuable data and 
findings to inform policy and prac- 
tice. It is hoped that the absence of 
mention is not an indication that such 
support will not be forthcoming in the 
third strategy. The study on perform- 
ance assessment of students’ I L skills 
reported earlier in this paper is only 
a preliminary study, and should be a 
priority area for further research. 
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Introduction 

The study ‘Indicators on ICT in edu- 
cation’ was run under the auspices of 
the European Commission. The study 
was run from November 2008 until 
October 2009 (’). The main purposes 
of the study were the following. 

1. To identify a set of indicators that 
are relevant for enabling the regu- 
lar monitoring of the use and impact 
of ICT in primary and secondary 
education. 

2. To describe scenarios for moni- 
toring ICT in education in the 
European Union. 

The study was focused on the 27 EU 
Member States, the three candidate 
countries and the countries from the 
European Economic Area. This group 
will hereafter be referred to as ‘EU+’. 
In line with the main steps for moni- 
toring that are described in Chapter II: 
Monitoring in education: an overview. 
The main questions addressed in this 
study were the following. 

1. What are the policy issues regard- 
ing ICT in education? 


(') This study was financed (at a cost of 
EUR 122 200) by the European Commission. 
Contract EACEA-2007-3278. Opinions presented 
in this chapter do not reflect or engage the 
Community. © European Commission 


2. In which areas are indicators 
needed? 

3. Which international comparative 
data are available and what are the 
data gaps? 

4. Which actions could be under- 
taken for ensuring that monitor- 
ing of European benchmarks and 
international comparisons on edu- 
cational progress will take place in 
the future? 

Each of these questions will be 
addressed in the subsequent sections. 

1. Policy issues 

As explained in Chapter II. educational 
monitoring is primarily a tool for 
policymaking and, hence, a first step 
in the process of exploring scenarios 
for monitoring ICT in education in 
the EU+ consisted of analysing the 
intentions of policymakers with regard 
to this area. A distinction will be made 
between common objectives and 
common goals/topics. The common 
objectives were inferred from EU 
policy documents reflecting common 
ICT-related objectives that originate 
from the Lisbon strategy and follow-up 
declarations. For all the EU+ countries 
that were targeted in this study, policy 
documents were collected from 
several sources, for instance official 
policy documents issued by ministries, 
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reports available through the EUN 
Insight project and/or articles about 
national policies regarding ICT written 
by researchers in a recent book edited 
by Plomp et al. (2009). A qualitative 
analysis of these documents was 
conducted, which resulted in a list of 
policy topics. 

In the next sections, these topics will 
be summarised and the issues under- 
lying these topics will be described. 
This will constitute the basis for forming 
an impression of the current relevance 
of indicator domains for these topics 
in the targeted group of countries for 
which a survey was conducted among 
ICT policy experts from the EU + coun- 
tries (see Section 1.2). 

7.7. EU policy topics regarding 
ICT in primary and 
secondary education 

At the EU level, several initiatives 
were taken to promote the use of ICT 
in education. With regard to ICT, one 
common EU objective resulting from 
Lisbon was that ‘schools and train- 
ing centres, all linked to the Internet, 
should be developed into multi-pur- 
pose local learning centres accessible 
to all, using the most appropriate meth- 
ods to address a wide range of target 
groups; learning partnerships should 
be established between schools, 
training centres, firms and research 
facilities for their mutual benefit'. This 
objective implies, for instance, that 
100 % of schools should have access 
to the Internet. 

In follow-up declarations, these objec- 
tives have been further elaborated in 
more specific terms. In a recommenda- 
tion of the European Parliament and of 
the Council of 18 December 2006 on 
key competences for lifelong learning 
(http://eur-lex.europa.eu/LexUriServ/ 


LexUriServ.do?uri=OJ:L:2006:394:00 
10:0018:en:PDF), eight main compe- 
tency areas were distinguished: 

communication in the mother tongue; 

communication in foreign languages; 

mathematical competence and 

basic competences in science and 

technology; 

digital competence; 

learning to learn; 

social and civic competences; 

• sense of initiative and 
entrepreneurship; 

cultural awareness and expression. 

These areas will be further referred to 
as ‘the EU core competency areas’. 
Although most of these competency 
areas can be considered more or less 
traditional (as they always tended to 
be featured in national educational 
goals of countries), some of these 
(such as ‘digital competence’, ‘learn- 
ing to learn’ and ‘sense of initiative 
and entrepreneurship’) are believed to 
be essential for the information soci- 
ety, but also an underlying expectation 
can be observed that ICT is a crucial 
facilitator for acquiring and maintain- 
ing competencies in these areas. 
‘Learning to learn' can be conceived 
as a basic skill underlying the abil- 
ity for lifelong learning, and, hence, 
against this background it is relevant 
to observe that in the Council’s con- 
clusions on a strategic framework for 
European cooperation in education 
and training for the period until 2020 
(‘ET2020’), the importance of lifelong 
learning is reiterated (http://www.con- 
silium.europa.eu/uedocs/cms_data/ 
docs/pressdata/en/educ/1 07622.pdf). 

The common objectives are not explicit 
in terms of performance expectations 
and ICT-related opportunities to learn. 
It seems fair to infer that the underly- 
ing assumption is that countries are 
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expected to implement opportunities 
for students that lead to improvements 
in these core competency areas. With 
regard to ICT, ‘access for all' implied in 
relation to these areas could be inter- 
preted as ‘opportunities for students 
in school to use ICT for learning’. 
However, concrete targets need to be 
further defined. 

The implications for our study are that, 
when it concerns students, competen- 
cies and attitudes will mainly refer to 
these EU core competency areas. 

1.2. National ICT-related policy 
topics for primary and 
secondary education 

An analysis of ICT-related policy docu- 
ments from the targeted group of EU+ 
countries was undertaken. All policy 
documents collected were read and 
coded, and the topics that were cov- 
ered in these documents were listed. 
This resulted in a long list of cat- 
egories that were classified in terms 
of main topics, sub-topics, sub-sub 
topics, etc. 

The main topics that resulted from this 
analysis were as follows. 

1 . Infrastructure: this concerns issues 
such as hardware and software 
and sub-issues such as access to 
the Internet, broadband connec- 
tions and open-source software. 

2. Curriculum and content: this 

covers issues such as pedagogical 
approach (e.g. autonomous learn- 
ing), content (e.g. development of 
methods), assessment (e.g. portfo- 
lios, digital drivers licence). 

3. Outcomes, e.g. competencies, dig- 
ital literacy. 

4. School leadership, e.g. change 
management. 


5. Connectedness, e.g. national and/ 
or international cooperation, pub- 
lic-private partnerships. 

6. Teacher training, e.g. teacher com- 
petencies, pedagogical drivers 
licence. 

7. Support, e.g. the way technical 
and/or pedagogical support is 
made available. 

8. Transversal issues, e.g. equity, 
financing, safety. 

In the next section, policy issues that 
are underlying these topics will be 
described. 

1.2.1 infrastructure 

Infrastructure as a topic is very broad. 
It covers sub-topics such as hardware 
and software, which are still quite 
broad, as policy concerns with regard 
to hardware cover a further wide range 
of topics, as is the case with software. 
The overall picture regarding policy 
issues with regard to infrastructure 
resulting from the analysis of policy 
documents can be summarised as 
follows. 

A first observation is that ICTinfrastruc- 
ture is still an important topic for policy 
concerns. This topic is addressed 
in almost all documents and can be 
considered a crucial condition for the 
use of ICT: ICT infrastructure should 
be present before ICT can be used. 
In the early days of the introduction of 
computers in education, a shortage of 
hardware and/or software was often 
mentioned by educational practition- 
ers as a major obstacle for integrating 
ICT in teaching and learning. 

The policy documents refer to the 
intention to improve the current infra- 
structure, namely: 
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equipping classrooms with fast 
Internet connections (e.g. Austria, 
Belgium); 

providing interactive white boards to 
schools (e.g. UK, Denmark); 
standardising systems and software 
(e.g. UK); 

providing laptops for teachers (e.g. 
UK); 

improving buildings (e.g. Cyprus); 
ensuring own e-mail addresses 
for students and teachers (e.g. 
France). 

It is interesting to note that some 
countries seem to take more initiatives 
than others regarding the provision of 
new equipment. For instance, the UK 
made substantial investments, while 
countries like Denmark (pilot project) 
and Sweden were more hesitant. 

In some countries, equipping schools 
no longer seems to be a policy pri- 
ority, for instance in Norway where 
‘there are no national programmes or 
initiatives for introducing new devices 
in schools’. This last observation is 
important, because it illustrates that 
countries are in different stages of 
introducing ICT in education. This will 
have consequences forthe monitoring 
needs of these countries. 

Several policy strategies are in place 
for allocating and financing equipment 
in schools, namely: 

lump sum financing in the 
Netherlands; 

in Estonia, a school must submit a 
statement indicating how it is cur- 
rently using ICT in its teaching and 
learning programmes; the school 
must also detail how it will use the 
new equipment; 

in Slovakia, the schools have to pre- 
pare a project proposal in which they 


present a vision of how they would 
use ICT in their schools. 

In Belgium and other countries, the 
access of students (and even the local 
community) to ICT infrastructure out- 
side school hours is stimulated. 

With regard to software, it can be 
noted that the developement and 
maintenance of high-quality software 
for education has been a challenge 
since the first micro-computers were 
installed in schools around the mid- 
1980s. Recent initiatives concern the 
creation of educational portals offering 
open content (via Internet-reachable 
databases containing educational 
content in many different forms) and 
the promotion of using open-source 
software which, in principle, can be 
attuned to the needs of the users (a 
common problem in education is that 
teachers do not like pre-cooked con- 
tent which they cannot change). In the 
UK, the curriculum online programme 
(see http://www.dfes.gov.uk/curriculu- 
monline/) was launched in 2003 and 
provided every teacher and school 
with e-learning credits that they could 
spend on approved ICT resources 
purchased through the website. 

1.2.2 Curriculum and content 

A curriculum allows governments to 
regulate (formally and prescriptively 
or less formally) educational proc- 
esses in order to influence outcomes 
of learning. Educational practitioners 
often mention the time that is needed 
for realising the existing curriculum as 
a major obstacle for implementing ICT 
in teaching and learning. In the UK, 
‘improving the quantity and quality of 
e-learning is irrelevant, however, if it 
is not done within the context of cur- 
riculum development’. 
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The policy documents analysed often 
(in more than 50 % of the documents) 
refer to curriculum measures that were 
planned in order to promote the inte- 
gration of ICT. Frequently mentioned 
are the intention to integrate ICT in 
school subjects and the development 
of methods for ICT-assisted learning. 

A major distinction that can be made 
is between learning about ICT (ICT as 
an object) and learning with the help 
of ICT (ICT as a tool). Whereas in 
some countries the acquisition of ICT 
skills is organised via a separate sub- 
ject, in other countries it is assumed 
that these skills can be acquired 
via the traditional subject areas (for 
instance, in some German states, it is 
integrated in media education, while in 
other countries, in particular the new 
member countries, a separate infor- 
matics subject exists). Some docu- 
ments are very explicit about the issue 
of separate ICT subjects, for instance 
Belgium where ‘the policy underpin- 
ning the plan is to incorporate ICT into 
different courses rather than to intro- 
duce a specific ICT-related course'. 

Next to the expectation that ICT can 
improve outcomes of learning in tra- 
ditional subject areas, a number of 
policy documents also mention that 
ICT can help to implement new ways 
of learning whereby the students (with 
the help of ICT) acquire more control 
and responsibility for their own learn- 
ing processes and outcomes. For 
instance, digital portfolios are con- 
ceived as a tool that can help to keep 
track of learning activities and prod- 
ucts resulting from these activities. 

Making explicit links between digital 
instructional materials and curriculum 
goals (the Netherlands) is conceived as 
helping the teacher to choose appro- 
priate ICT applications. In the UK, 


mention is made of ‘learning design 
packages that would enable teachers 
in all sectors to build their own individ- 
ual and collaborative learning activi- 
ties around digital resources'. 

In some countries, the intention is 
explicitly formulated that ICT should be 
a daily part of student learning activi- 
ties (e.g. Belgium, Estonia), which is 
an example of an explicit educational 
policy objective dealing with promot- 
ing the opportunities of students to 
learn with and/or about ICT. 

1.2.3 Outcomes 

From the policy plans a clear expec- 
tation is transmitted, phrased in differ- 
ent terms and with different degrees 
of explicitness, but with an underlying 
strong conviction that the use of ICT 
in education can improve access to 
teaching and learning opportunities, 
help to enhance the quality of teach- 
ing and learning, improve learning out- 
comes and promote positive reform of 
education systems. 

Flowever, these expectations are 
global in character. The past decades 
have witnessed a search for getting a 
better insight into what impact may be 
expected from applying ICT in educa- 
tion. It is still unclear how to answer 
questions, such as the following. 

1 . What are the basic functional e-lit- 
eracy skills that students should 
master when they leave compul- 
sory education? 

2. In which content areas can most 
added value be expected when 
ICT is applied? 

3. Flas the use of ICT in the past dec- 
ades improved the competencies of 
our students in core subject areas? 
Are students better prepared for life- 
long learning (in terms, for instance, 
of motivation to learn, analysing 
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their own shortcomings, setting out 
learning trajectories, self-assess- 
ment, problem solving, etc.)? 

In this respect, a recent ‘knowledge 
mapping’ exercise conducted by the 
World Bank’s /nfoDev Group (Trucano, 
2005) is relevant. It revealed that, 
despite decades of large investment in 
information and communication tech- 
nologies to benefit education in OECD 
countries and despite the increasing 
use of ICT in education in develop- 
ing countries, data to support the per- 
ceived conviction on the benefits from 
ICT are limited and evidence of effec- 
tive impact is very elusive or debat- 
able. These findings highlighted vari- 
ous knowledge gaps and recognised 
the need for internationally accepted 
standards, methodologies and indica- 
tors to better measure the real benefits 
of ICT in education. 

This lack of good quality and unques- 
tionable data, in addition to the 
absence of standardised guidelines 
for establishing relevant and compa- 
rable indicators, hinders the ability of 
policymakers to make informed deci- 
sions or to demonstrate greater volun- 
tarism towards the integration of ICT 
into their education systems. 

The above is not meant to claim that no 
research has yet been done regarding 
these questions. Many research and 
meta-studies have been conducted 
over the past decades. Most of these 
studies, however, do not deal with 
changes in the total education system, 
and, therefore, when policymakers 
have to take policy initiatives for the 
educational system at large, they 
often stand with empty hands. The 
policy documents that were analysed 
offer the following expose regarding 
(expected) outcomes. Objectives of 
new or revised curricula for primary 


and secondary education should also 
pay attention to ICT-related competen- 
cies, sometimes combined with media 
literacy (among others, Germany). For 
this purpose, ICT can be a separate 
subject or integrated in other subject 
areas. Several countries made a delib- 
erate choice for either one of these 
models. However, countries differ with 
regard to what is included in the ICT 
competencies. Some countries have 
examinations to establish these com- 
petencies, such as the junior compu- 
ter driver’s licence. A prerequisite is 
that all students have the opportunity 
to use ICT during their schooling or at 
home. The latter refers to, for example, 
disadvantaged students in second- 
ary education (United Kingdom). The 
efforts countries undertake to include 
ICT in the curriculum fall under the 
umbrella of the more general goal of 
bridging the digital gap by providing all 
citizens with opportunities to acquire 
basic ICT skills and skills to use all 
kinds of ICT services. 

Another goal is that students are 
well prepared for the labour market. 
Governments of some countries initi- 
ated programmes to promote access 
to computers and the Internet at 
home. Students get an extra oppor- 
tunity to use a computer and to learn 
with computers. It is not only students 
who benefit from these programmes, 
but also their families. 

1.2.4 School leadership 

For a long time (since the introduction 
of the first micro-computers in educa- 
tion) the issue of school leadership 
was not featured in many policy plans. 
However, it seems that (probably as a 
result of diffusion of research results 
regarding mechanisms that play a role 
in successful educational changes) 
awareness is increasing that school 
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leaders may be important gatekeep- 
ers and facilitators in the implementa- 
tion of ICT. 

We extracted the following observations 
fromthe policy documents. School lead- 
ers need appropriate training in a new 
kind of management in which ICT is a 
permanent factor from now on in their 
strategy. In the UK, tools are provided 
to help school leaders to assess how 
well their organisation uses ICT. These 
tools help to modernise the school 
management (Austria). In Belgium, 
school leaders have to develop their 
ICT policy instead of using an imposed 
policy document made by the gov- 
ernment. School leaders also have 
to do this in Germany. The reason is 
that they can describe their vision but 
also become aware of what is needed 
to achieve this vision and the impact 
on teaching and learning. Norwegian 
schools are required to develop an ICT 
plan. An ICT policy document is not 
required in Sweden though local stake- 
holders ask school leaders to have 
one. Each school has to make a quality 
report every year. This report includes 
plans for how to improve. Schools in 
Malta have included their ICT policy in 
their school development plan. One of 
the topics that has been identified in the 
research literature as important when 
it concerns school leadership is the 
development of a common vision on 
ICT that is shared by all stakeholders 
in the school (and preferably consist- 
ent with the vision from stakeholders 
outside the schools, such as ministry, 
inspectorate, parents). This topic is 
hardly addressed in the policy docu- 
ments that were analysed. An excep- 
tion is the UK where BECTA aims 'to 
deliver a vision for ICT in schools'. 

1.2.5 Connectedness 

ICT can help to open the school to the 
world as well as vice versa by allow- 


ing the real world to enter the school 
more easily. The walls of the school 
and the classrooms are no longer dif- 
ficult blockades for integrating real-life 
components in the learning process. 
There is also a growing awareness 
that ICT innovations within schools 
cannot be realised without the help of 
the outside world and that the help of 
outside colleagues and even business 
firms is needed. 

In the policy documents, we find this 
reflected in several examples in almost 
all European countries. Most important 
are the links between schools and pri- 
vate partners (business companies). 
Several companies in the field of ICT, 
such as Apple, Intel and Microsoft, 
are involved in partnerships. For the 
schools in the respective countries, 
the public-private partnerships involve 
training of teachers, development 
of ICT-related educational materi- 
als (including e-learning and portals), 
infrastructure (hardware and access to 
the Internet) and support and/or fund- 
ing. Most of the public-private partner- 
ships are taking place at national level, 
but some are regionally based, as in 
France. In several European countries, 
projects have been set up to establish 
a link between school and the school 
environment. These projects vary in 
their goals: enabling students to learn 
at home or in hospital, informing par- 
ents of the achievements of their chil- 
dren and to have contact with teachers, 
increasing digital literacy of other family 
members (including parents) or provid- 
ing access to the Internet at home. 

1.2.6 Teacher training 

Before teachers can apply ICT in 
their lessons, they first need to know 
what ICT is and how it may be used 
for improving instructional processes. 
Hence they need to be trained. It is dif- 
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ficult to contradict a statement like this, 
but it is even more difficult and quite 
often impossible to realise adequate 
continuous staff development activi- 
ties for all teachers in an education 
system. Since the early days of ICT in 
education, policy solutions have been 
tried in order to train teachers ade- 
quately but the complaints about the 
lack of teachers' competencies and 
confidence remained and hence the 
search for adequate solutions (that 
are also payable) is continuing. Many 
promising initiatives were taken, and 
applied in small contexts, but were 
probably not upscaleable. 

The current policy issues that were 
inferred from the policy documents are 
summarised below. In all European 
countries, the in-service training of 
teachers is a policy issue. Training 
programmes and other arrangements 
have been set up to organise the train- 
ing of teachers. Teachers are offered 
opportunities to learn how to use ICT 
fortheirown use and howto use itinthe 
teaching-learning process. An example 
is Hungary, where ‘teacher training is 
beginning to concentrate on ICT-based 
educational methodology, with particu- 
lar emphasis on how to make optimal 
use of educational technology in the 
classroom’. Several countries have for- 
mulated ICT competencies for teach- 
ers, including the didactical skills to use 
ICT in the classroom (‘using ICT as a 
pedagogical tool’). In some countries, 
the training results in a certificate or the 
European computer driving licence. In 
Lithuania, for example, the basic mod- 
ules of the European computer driving 
licence have been extended with addi- 
tional modules specifically related to the 
use of ICT in schools, as in Denmark, 
where the pedagogical computer driv- 
ing licence has been developed. In 
some countries, such as the United 
Kingdom and Lithuania, attention is 


also paid to the training of librarians in 
the field of ICT. 

Within the framework of ICT projects, 
programmes have been set up for the 
in-service training of teachers, among 
others the MoNES programme in 
Poland, the KK-foundation in Sweden, 
the POCTI programme in Portugal, 
FOR TIC in Italy, Infovek in Slovakia 
and OPE.fi in Finland. In many coun- 
tries, the teacher training institutes are 
involved in the in-service training of 
teachers. One would expect that, next 
to the in-service training of teachers, 
the European countries consider the 
pre-service training of teachers as an 
important issue. However, only a few 
policy (related) documents state this 
issue. In Belgium, the institutes for 
teacher training have to pay attention to 
the ICT competencies of their students 
by setting new attainment targets and 
goals, not only for the basic ICT skills 
but also for skills related to using ICT in 
the teaching-learning process. 

1 .2.7 Support 

At the beginning of the computer era, 
technical support in schools was impor- 
tant. Teachers did not have sufficient 
ICT knowledge to solve hardware and 
software problems. At that time, hard- 
ware and software in schools were less 
reliable. Nowadays, technical support 
is provided quite often by professionals 
or well-trained staff in schools, espe- 
cially in secondary education. 

Even more important than techni- 
cal support is pedagogical support 
needed by teachers when applying 
ICT in teaching and learning. The 
policy documents that have been 
reviewed show that teachers may 
have difficulties in implementing ICT 
in the teaching-learning process and 
that they need support to accomplish 
this task. Mostly, the support is pro- 
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vided by agencies outside the school. 
In Sweden, ‘Schoolnet offers many 
different services, functioning as an 
information centre, a library and a 
news agency. Schoolnet provides a 
platform for the development of new 
educational approaches opened up by 
the Internet and new multimedia tech- 
nologies.’ In Portugal, the Ministry of 
Education relaunched the Nonio pro- 
gramme to broaden the ICT compe- 
tence centres network to support all 
school groups in the country. Hence, 
there are indications of a change of 
emphasis from technical support to 
pedagogical support. This is, among 
other things, reflected in the role of 
school ICT coordinators, which in 
some countries is no longer limited to 
technical support. Educational sup- 
port, including in-service training, is 
a task of the coordinator. In Catalonia 
(Spain), ‘a new job description for ICT 
coordinators in schools (with specific 
regard paid to the new breed of tech- 
nical support services), reforming in- 
service teacher training and setting up 
new pedagogical support services for 
ICT using personnel from pedagogical 
resource centres’ has been created. 
In some countries (e.g. Portugal), the 
function of ICT coordinator does not 
exist and, hence, the teachers have to 
organise the technical and pedagogi- 
cal support in their schools. 

1.2.8 Transversal issues 

In the documents, there are a number 
of recurring issues that can be consid- 
ered transversal, as they cut through 
the categories that were described 
above. A number of these issues are 
reviewed below. 

Equity 

Almost all countries have the policy that 
all citizens should have equal opportu- 
nities in society. It is expected that the 
use of ICT can foster these chances. 


Therefore, courses in basic ICT skills 
are set up or people are given access 
to ICT facilities after office hours. 
Activities take place within the frame- 
work of digital literacy for all, narrowing 
the digital divide and lifelong learning. 
A policy goal in Finland is that all citi- 
zens have ‘opportunities and the basic 
capabilities to use electronic services 
(e-services) and content’. 

Special programmes are aimed at cer- 
tain groups in society: disadvantaged 
children in (secondary) education, 
students who are ill, young sportsmen 
and sportswomen, young migrants or 
certain regions in a country. Several 
programmes also focus on parents and 
other groups (elderly persons, disabled 
persons). The programmes provide 
training in basic skills, access to (broad- 
band) Internet, computers at home or 
digitalisation of (learning) materials. 
Disabled persons are often faced with 
ill-adjusted standards and extra costs 
for hardware. This limits their access to 
the knowledge society. 

Documents from Sweden and Portugal 
state that there is no specific pro- 
gramme in these countries. 

Financing 

Governments (mostly ministries of 
education) in several EU countries 
purchase the hardware, software 
and access to the Internet and/or 
they finance the training of teachers. 
Sometimes local governments are 
involved too, as in Poland. 

Initially, hardware was financed by 
grants and sponsors in Slovakia 
because the government had not yet 
set up an information technology pro- 
gramme. Programmes were later set up 
to give schools access to the Internet. 
By participating in European projects, 
schools received equipment. 
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Safety 

In policy documents, two aspects are 
distinguished regarding safety of ICT 
use: first is the protection of children 
against harmful content; second the 
critical evaluation and use of sources. 
For instance, in Sweden, a programme 
was set up to raise the awareness of 
children, parents and educators with 
regard to the first aspect. Other coun- 
tries have started similar initiatives. 

In Malta, a portal has been devel- 
oped to protect schools from inappro- 
priate content and it also offers links 
to useful educational websites. The 
Greek school network has a protec- 
tion policy for students. In many coun- 
tries, protection is often offered by the 
government through providing filter- 
ing techniques, information on how to 
use the Internet or a telephone line to 
report illegal information. Campaigns 
have been launched to increase citizen 
awareness. 

Monitoring 

In order to be able to evaluate ICT 
policy in education, monitoring of the 
implementation takes place in quite a 
number of countries. 

The monitoring focuses, among other 
things, on infrastructure, competen- 
cies, integration in the teaching-learn- 
ing process, perceptions, attitudes and 
needs. 

1.2.9 From policy issues to 
conceptual framework 

The policy topics mentioned above 
may be conceived as concepts that 
can constitute the basis for a concep- 
tual framework in which expectations 
about interdependencies can be made 
explicit. Although the policy docu- 
ments are not usually very explicit 
about cause and effect expectations, 


such expectations certainly exist. For 
instance, one of the many possible 
conceptualisations of expected rela- 
tionships is shown in Figure 1, which 
contains most of the issues that were 
identified in the document analysis 
and can be summarised as follows. 
The ICT learning opportunities of stu- 
dents have a (hypothesised) impact 
on the competencies and attitudes 
that they acquire. These opportunities 
are believed to depend on the peda- 
gogical practices of teachers (which 
in turn depend on the extent to which 
the teachers are trained) and availa- 
bility and access to ICT infrastructure, 
which is a crucial condition for creating 
ICT-OTL at school. On the other hand, 
these opportunities are determined by 
what students learn outside school. 

Policymakers can influence these con- 
ditions via curricula, but countries differ 
in the extent to which the curricula can 
be prescriptive. As the use of ICT is an 
educational change, the role of school 
leaders is important as well as the 
availability of teacher training facilities 
for getting acquainted with the techni- 
cal and pedagogical aspects of ICT. 

The fact that the policy documents are 
not specific with regard to expected 
ICT-OTL and impact is not surprising, 
on the one hand, as it is currently still 
too early to take policy decisions for 
the education system at large, since 
it is not yet known what works and 
what does not beyond the borders 
of small-scale pilots, case studies, 
experiments and the like. On the other 
hand, although policy documents are 
usually not very specific, given the 
large investments with regard to ICT 
infrastructure in education, one would 
expect more explicit expectations to 
be formulated. However, if clear policy 
expectations are lacking, one may 
wonder what implications this may 
have for EU monitoring. 
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Figure 1 : The main concepts for monitoring ICT use and impact 


1.2.10 Implications for 

monitoring the use and 
impact of ICT in the EU 

As mentioned in Chapter 2, in order 
to be able to make inferences about 
whether progress is being made with 
regard to educational outcomes, poli- 
cymakers need monitors that show — 
on the basis of reliable and valid quan- 
titative indicators — to what extent 
expected changes are taking place over 
time. Although the analysis of policy 
documents does not immediately lead 
to identifying common goals, they offer 
a first step for delineating goal domains 
that can (in principle) be further defined 
and used for an exploration among ICT 
policy experts from the EU+ group of 
countries. The approach for this explo- 
ration is described in Section 1.3. Next 
to finding empirical evidence for the 
relevance of indicator domains, the 
common objectives (resulting from 
the Lisbon strategy) can (in principle) 
be used as a basis for more concrete 
indicator definitions. However, as 


described in Section 1.2.1, these need 
to be further specified in order to be 
useful for drawing up indicator defini- 
tions and operationalisations. 

1.3. Percei ved re leva n ce 
of indicator areas 

A panel of 54 national ICT policy 
experts from the EU+ countries were 
invited to give their opinion about the 
need for international comparative 
ICT indicators in 55 indicator areas 
that covered the policy topics men- 
tioned in Section 1.2. The question- 
naire was administered online in the 
period May/June 2009. Responses 
were received from 76 % of the invited 
experts from 26 countries (which is 
93 % of the countries with which active 
communication channels were estab- 
lished). The data were processed with 
SPSS-16. If more than one response 
was received from a country, the 
responses were weighted so that in 
the end result each country had the 
same weight. In the following sections, 
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the results from this survey will be 
summarised. 

Firstly, a description will be given of 
the extent to which the respondents 
experienced in general a need for com- 
parative indicators on ICTin education. 
Next, an overview will be given of the 
areas for which the highest needs were 
expressed. 

Several caveats should be taken into 
account when using the ratings pre- 
sented in the next sections for setting 
indicator priorities. Firstly, the descrip- 
tions for each area were quite general 
and hence more concrete indicator 
elaborations could elicit different indi- 
cator needs, as usually is the case: the 
more concrete a proposal, the less con- 
sensus may be expected among panel 
members. Also, one should take into 
account that the ratings concern sub- 
jective estimates of panel members, 
which do not necessarily reflect the 
opinions of national educational actors 
involved in decision-making about edu- 


cational matters and in particular what 
to monitor, how extensively and how 
frequently. Nevertheless, the ratings 
can be used for a first priority list which, 
when it eventually comes to monitoring 
ICT in the EU, can be further revised 
in subsequent negotiations between 
countries, taking into account too areas 
other than the ones considered in our 
study. 

1.3. 1 The need for comparative 
indicators in general 

A first question for which opinions were 
solicited from the panel members con- 
cerned the need for international com- 
parative monitoring in the EU of ICT in 
education. From Figure 2 we can infer 
that, among the panel members, there 
was a high consensus. Slightly more 
than 50 % of the respondents are defi- 
nitely sure that this need exists, while 
another 38 % think that this is the case 
depending on the kind of indicators. 
Hence, altogether, a large majority 
(92 %) indicated that there is a need for 



Percentage of countries 


Figure 2: The need for ICT monitoring in general 
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international comparative monitoring of 
ICT in education. In only one country 
did the panel member have the opinion 
that this need did not exist. Apparently, 
the panel members felt confident about 
their capacity to rate, because the 
answer category ‘Don’t know’ was not 
used at all. It could be inferred from 
this poll that, throughout Europe, there 
is a need for international comparative 
monitoring of ICT in primary and/or 
secondary education. 

1.3.2 High priority indica tor areas 

Keeping in mind the caveats expressed 
earlier in this chapter, an indication 
of priority areas for monitoring may 


be given by taking 60% agreement 
about high needs as the threshold for 
selecting indicator areas. The indicator 
areas which 60% or more of the panel 
members qualified as highly needed is 
shown in Table 1 . 

The issues ‘Connectedness’, 
‘Curriculum and content’ and ‘Infra- 
structure’ did not contain topics for 
which 60 % or more of the panel mem- 
bers expressed a high need. The fact 
that ‘Curriculum and content’ was not 
rated as highly needed by many panel 
members is a bit surprising, as it is often 
argued that the curriculum is an impor- 
tant handle for introducing educational 
change. 


Table 1: Indicator areas per topic with high need above 60 % 


Indicator areas 

Percentage 
high need 

a. 

Opportunities to learn with and/or about ICT 



Activities where students use ICT for learning in the 5 EU key competency 
areas (literacy in reading, mathematics and science, language skills, ICT skills 
and learning to learn skills) 

73 % 



The extent to which students use ICT for cooperation and/or communication 

65 % 



Activities where students use ICT in general at school 

60% 



The extent to which students use ICT at school 

60% 

b. 

Competencies and attitudes of students 



The ability of students to solve assignments that require the use of ICT in the 
5 EU key competency areas (literacy in reading, mathematics and science, 
language skills, ICT skills and learning to learn skills) 

66 % 



The ability of students to use ICT for learning to learn (goal setting, self eva- 
luation, management of learning, self evaluation) 

64% 

c. 

ICT support 



The extent to which pedagogical support is available for teachers (for lesson 
preparation, class management issues, assessment procedures, etc.) 

61 % 

d. 

Teacher training 



Pedagogical ICT competencies of teachers 

82 % 



Ability of teachers to build their own individual and collaborative learning 
activities around digital resources 

68% 



Ability of teachers to locate digital content resources that fit their curriculum 
targets 

62 % 



Application of innovative forms of assessment 

61 % 

e. 

School leadership 



Competencies of the school leadership to manage ICT-related innovations 

63 % 
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2. ICT-related data 
available in regular 
assessments from the 
IEA and OECD 

In order to determine which data 
and instruments with regard to ICT 
were available in the regular assess- 


ments from the OECD and/or IEA 
which have been conducted since 
2000, all questionnaires from these 
studies were collected and mapped 
on the list of policy topics that were 
described in Section 2.2. The ICT- 
related data which are available in 
the existing data sets are listed in 
Table 2 below. 


Table 2: ICT data available in data bases from IEA and/or OECD 


Infrastructure 

Primary education 

Secondary education 

— Number of computers available for 

— Availability of computer software at 

instruction (school leader) 

students' home 

— Number of Internet computers at school 

— Availability of computer at students' 

— Shortage of computers for instruction in 

home 

general (perceived by school leaders) 

— Access to Internet at students' home 

— Shortage of computers for instruction in 

— Access to Internet in general (teachers) 

mathematics/science (school leaders) 

— Access to Internet for mathematics/ 

— Access to Internet in general (teachers) 

science (teachers) 

— Access to Internet for mathematics/ 

— Shortage of computers for instruction 

science (teachers) 

mathematics/science (teacher) 

— Computers available in classroom and/or 

— Computers available for mathematics/ 

elsewhere (teacher) 

science (teacher) 

— Computers available for educational 

— Shortage of software for mathematics/ 

purposes (teacher) 

— Computers available for mathematics/ 

science (teacher) 

science (teacher) 


— Availability of computer at students' 
home 



Use reported by students 


Primary education 

Secondary education 

— Computer use in general 

— Computer use in general 

— Computer use at school 

— Computer use at school 

— Computer use outside school 

— Computer use in mathematics 

— Internet use outside school 

— Computer use outside school 

— Use of computers for communication 

— Use of Internet outside school 

purposes 

— Use of Internet at school for: 
Downloading music 
Collaboration 

— Use of computers for: 
Playing computer games 
Writing stories or reports 
Spreadsheets 
Graphical software 
Programming 
Downloading 
Searching information 
Communication 
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Support 

• Primary education 

— Availability of educational support 
(perceived by school leaders) 

— Shortage of technical support (perceived 
by school leaders) 

— Person who is providing educational 
support 

Secondary education 
— Shortage of support for mathematics/ 
science (perceived by teachers) 


In addition to the above, the OECD 
databases also contain data about the 
years of experience in computer use 
that students had at the time of data 
collection. 


For the purpose of our study available 
statistics about students' use of ICT 
and infrastructure were extracted from 
the available data bases. The statis- 
tics that are included in the final report 
of this project are listed in Table 3. 
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For space considerations, in the next 
section only a limited number of statis- 
tics listed in Table 2 are shown. 

2. 7. Statistical indicators from 
existing assessments: what 
do they show? 

The review of the available interna- 
tional comparative ICT indicators on 
students’ use of ICT and infrastructure 
revealed the following. 

Many data gaps exist, for instance 
for some EU+ countries data are 
completely lacking and for many 
others the time series since 2000 
are not complete. 

Some indicators have reached the 
end of their lifetime. 

Some results are unexpected and 
more in-depth validity research is 
needed. 

• International comparative data (and 
associated measurement instru- 
ments) regarding the core areas that 
should be the focus of monitoring 
ICT do not exist. 

Below, a few statistics illustrating the 
observations mentioned above will 
be shown for primary education. The 
reader is referred to the final report for 
a more comprehensive description. 

An example of data gaps as well as 
lifetime can be observed in Figure 3 
which looks at the question of whether 


students at grade 4 primary education 
level ever used a computer at all. This 
indicator is based on questions shown 
in Box 1. A first observation from 
Figure 3 regards the data gaps with 
regard to the coverage of EU+ coun- 
tries and the incomplete time series. 

For most countries for which data 
existed from 2007 the conclusion 
seems warranted that nearly all stu- 
dents in primary education had used 
a computer at least once. Steady 
increases occurred from 2001 (some- 
times exceptional as in Latvia). The 
cross-study trends have face validity 
to the extent that an expected steady 
increase is indeed observed. 

An illustration of an observation that 
requires further in-depth research 
concerns the statistics for Italy where 
the percentage in 2007 is lower than 
in 2003 (a similar phenomenon was 
observed in the household survey 
from Eurostat). This could point to, 
although not necessarily, incompara- 
bility of samples. 

Another interesting observation is that 
this indicator has reached the end of 
its lifetime, because it is close to the 
ceiling of 100% (already even by 
2001 in some countries). This is due 
to the global character of the indicator 
(‘whether computers were used ever’) 
which had value in the early days of 
the introduction of computers, but 


Box 1 : Source of indicator 


Source: 

PIRLS2001 , TIMSS2003, TIMSS2007 

Question: 

Do you ever use a computer (do not include Nintendo, Gameboy or other 
TV/video game computers)? 

Answers: 

Yes, no 

Calculation: 

Percentage of yes answers 
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Figure 3: Percentage of grade 4 students having ever used a computer 


Percentage students having used computers at all, grade 4 



Sources: PIEA2001 : the IEA PIRLS assessment (reading) conducted in 2001 . TIEA2003 and TIEA2007: the IEA TIMSS assessment 
(mathematics and science) conducted in 2003 and 2007. For the meaning of country acronyms, see Annex A. 


currently it is more appropriate to 
zoom in on the intensity of use of 
ICT in general by students. Hence, 
in Figure 4 below, the percentages 
of students are shown who indicated 
that they used computers at least 
weekly. The calculations are based 
on a questionnaire item shown in 
Box 2. 

From Figure 4 one may infer that, 
in some countries, the weekly use 


of computers by students in grade 4 
substantially increased between 2001 
and 2006, particularly in countries 
that joined the EU more recently (for 
instance, Bulgaria, Latvia, Lithuania). 
In other countries (e.g. the Netherlands 
and UK), this statistic is reaching a 
ceiling and, hence, future statistics 
can be better expressed in terms of 
daily use of computers, perhaps with 


Box 2: Source of indicator presented in Chart 2 


Source: 

PIRLS2001 , PIRLS2006 

Question: 

How often do you use a computer in each of these places? 
At home, at school, other place. 

Answers: 

Every day or almost every day, once or twice a week, once or twice a 
month, never or almost never 

Calculation: 

Percentage students answering every day or almost every day or once or 
twice a week on use at home or use at school or use at another place 
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Figure 4: Percentage of grade 4 students using computers weekly overall 



Sources: PIEA2001 and PIEA2006: the IEA PIRL5 assessment (reading) conducted in 2001 and 2006. For the meaning of country 
acronyms, see Annex A. 


a further differentiation towards the 
number of hours per day. 

The few examples above concerned 
students' use of ICT irrespective of the 
context (inside or outside school) and 
the indicators show that most students 
are engaged with ICT and hence, in 


principle, there are ample opportuni- 
ties to learn with and/or about tech- 
nology. The question is whether stu- 
dents, in general (both inside as well 
as outside school), use computers 
for school work. This question has 
been addressed in TIMSS2007 (see 
Box 3). 


Box 3: Source of indicator presented in Figure 5 


Source: 

TIMSS2007 

Question: 

How often do you use a computer for your schoolwork (in and out of 
school)? 

— In mathematics 

— In science 

Answers: 

Every day, at least once a week, once or twice a month, a few times per 
year, never 

Calculation: 

Percentage students answering 'every day' or 'at least once a week' or 
'once or twice a month' 
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Figure 5: Monthly use in general for mathematics and science schoolwork, grade 4 



Sources: TIEA2007: the IEA TIMSS assessment (mathematics and science) conducted in 2007. For the meaning of country acro- 
nyms, see Annex A. 


The statistics in Figure 5 show that 
in most countries large groups of pri- 
mary school students do not seem to 
encounter opportunities for learning 
mathematics and science with the help 
of computers (either inside or outside 
school). This not only points to the exist- 
ence of digital divides in the population 
of students, but also to underuse of ICT 
in areas where many good examples of 
ICT applications exist. 

2.2. Reflections about available 
data 

From the previous sections one may 
infer that certain indicators have 
reached the end of their lifetime. This 
is, for instance, the case concerning 
the use of ICT on a daily basis by stu- 
dents. This indicator has witnessed 
major changes since the start of the 
current millennium, and it clearly 
shows that ICT is used in the daily 
life of students. Flowever, it was also 


noted that ubiquitous use of ICT in 
schools is still rare. One may wonder 
whether this should be judged nega- 
tively. Rather, the question emerges 
‘and so what'? As long as it is not 
known whether students’ skills are 
seriously hampered by a lack of ICT 
use in schools, this question cannot 
be answered. Hence, a plea should 
be made for measuring the extent 
to which students lack skills which 
evidently can be improved by more 
sophisticated use of ICT in teaching 
and learning. For planning future mon- 
itoring, this implies that the focus (as 
used to be the case in the past) should 
shift from monitoring ICT-related con- 
ditions (as was, for example, the case 
in SITES2006) to ICT-related student 
outcomes. This implies substantial 
investments in designing adequate 
instruments. With political will this 
should be possible: if mankind is able 
to create instruments to measure the 
characteristics of distant planets, it 
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is certain that, with adequate invest- 
ment, it should be possible to offer 
educational actors the instruments to 
observe what is happening in educa- 
tional practices. 

3. Recommendations 

Indicators for ICT-related student 
outcomes will have to be developed. 
International organisations (the EU, 
OECD, Unesco) could stimulate this 
development through their regular 
research programmes. A first step 
could be to generate frameworks for 
ICT use in the most important core 
competency areas and to create for 
each of these areas item banks con- 
taining concrete performance tasks 
that are perceived as relevant by a 
substantial number of countries. If, 
in the short term, the development 
of concrete performance tasks is too 
complex, it is advised to focus at first 
on definitions of these tasks and to 
monitor the extent to which students 
have opportunities (in and outside 
school) to acquire the competencies 
required by these tasks. In relation 
to this, it is recommended that inter- 
national organisations coordinate 
the development and elaboration of 
frameworks for monitoring. For the 
developers of indicators for the other 
areas, it is recommended that the indi- 
cator definitions are tuned to the com- 
petency frameworks. 

It is recommended that international 
organisations stimulate the creation 
and use of a worldwide instrument 
bank containing measures that can be 
used for assessing the development 
of ICT in education. Substantial pri- 
orities could be based on the overview 
provided in Table 1. Incentives might 
for instance consists of co-financing 
national projects in which measures 
from this instrument bank are used. 


The profit for countries consist of being 
able to use measures that have rela- 
tively high quality and are extensively 
tested, whereas where other countries 
use the same measures, compara- 
tive data also become available with- 
out the need for a heavy international 
overhead. 

It is recommended that studies are 
undertaken in which the characteris- 
tics and impact of existing ICT-related 
school monitors are investigated. 

It is recommended that interna- 
tional organisations coordinate their 
efforts to develop a vision regarding 
the future of monitoring educational 
change (of which ICT is one compo- 
nent). For the EU, a key question is 
whether this monitoring will be run 
fully under the auspices and control 
of the Commission addressing the EU 
core competency areas. 

This would be a vision for the long 
term (10-15 years) which could set 
the scene developing appropriate 
solutions for organisational, financial 
and methodological issues. Several 
elements that have been dealt with 
in this chapter (and Chapter II) could 
be part of such a vision, such as 

(a) capitalising on highly innovative 
forms of monitoring (through online 
data collection and authentic tasks), 

(b) holistic and multi-level monitor- 
ing (e.g. including school monitoring) 
and (c) tailored monitoring allowing 
for flexibility according to the indicator 
needs of countries. Part of this vision 
would be to sketch the responsibilities 
and roles of the different international 
organisations involved in regular inter- 
national comparative assessments. In 
the short term, the EU (but maybe this 
is also applicable to APEC and other 
organisations) could embark on exist- 
ing assessments that are run by OECD 
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and IEA in order to explore which 
desirable indicators can be included in 
these assessments and which options 
are feasible for guaranteeing an ade- 
quate geographical coverage of the 
EU Member States. 

4. Summary 
and discussion 

This article started with questions about 
monitoring ICT in education. It seems 
that clearly a need for monitoring ICT in 
education exists. But what then should 
be monitored? The main policy issues 
were identified in this article and the 
existence of international comparative 
ICT indicators was reviewed. It was 
argued that what ultimately counts in 
education are the skills and perform- 
ances of students. The overarching 
question is: Are students well enough 
prepared during compulsory schooling 
to adequately function in the informa- 
tion society? The answer, as implied by 
the previous sections, is that we do not 
have sufficient international compara- 
tive data available to address this ques- 
tion. At the moment we are inclined to 
monitor conditional factors, but this 
leaves open the question: 'What is 


wrong with the students’ skills for which 
ICT could offer solutions?’. 

An implication of our study is that, in 
years to come, intense efforts need 
to be undertaken to define 21st cen- 
tury skills, and the opportunities that 
schools should offer to students to 
learn with and about ICT. This calls for 
international cooperation, as it implies 
a substantial investment in the devel- 
opment of new curricula and assess- 
ment methods, which would probably 
outstrip the manpower and financial 
capacities of individual countries. 
What then is the role of the European 
Commission to ensure that appropri- 
ate and efficient methods for moni- 
toring will ultimately be in place? In 
this respect many potential actions 
could be considered of which the 
most prevalent ones were presented 
in Section 1.3.2. Still, the future tra- 
jectory is paved with uncertainties as 
much internal EU and external nego- 
tiation with third parties will be needed 
before a workable operational plan 
can be made. Nevertheless, the mes- 
sage appearing from our study is that 
the Commission has a very important 
potential role in stimulating and facili- 
tating these future developments. 
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I. Introduction 

At the start of the 21st century, human 
society is facing an information and 
communication revolution, resulting 
in the advent of new technologies. 
Computers and network technology 
have influenced a range of societal 
and cultural aspects of life as well 
as individual experiences. People in 
modern societies have different life- 
styles, thinking styles, ways of work- 
ing and new communication patterns 
compared to previous societies. This 
has been well proven by a variety of 
research findings in human and social 
science studies. Many enquiring 
scholars and practitioners have made 
an effort to discover the effects of 
technologies on individuals' lifestyles 
and communication modes. It may 
be assumed that different lifestyles 
result in different learning styles and 
outcomes. Some authors claim that 
digital technologies could be powerful 
transformational tools in individuals’ 
learning and growth. Even commer- 
cial videogames could have a positive 
impact on cognitive development and 
skills. Some other studies present the 
negative influence of technology use 
on human behaviours (Meyo, 2009). 
Even though there are inconsistent 
findings on the impact of advanced 
technologies in human life, no doubt 


is posed on the imperative for the 
effective use of digital technologies in 
education. 

Many efforts have been made to 
adopt information and communication 
technologies (ICT) to promote learn- 
ing excellence in various educational 
settings. At national and institutional 
levels, educational policies and regula- 
tions have been established to support 
the educational use of ICT. In schools 
and classroom settings, teachers and 
school administrators are attempt- 
ing to find the best ways to use ICT 
technology for their teaching and stu- 
dents’ success. However, accomplish- 
ments that are convincingly the result 
of the direct causal impact of ICT use 
are not always easily identifiable. It 
is even hard to ascertain the impact 
of ICT use in a simple way, because 
many other factors besides ICT itself 
might influence the ICT use in the indi- 
vidual’s genuine growth in education. 
Suppose that a 10th grader performed 
better in mathematics after using ICT 
in maths classes for a certain period 
of time. Of course ICT is an important 
tool for the student to improve his/her 
maths performance, but there might 
be other factors improving the per- 
formance, such as the way in which 
he/she uses ICT, learning contents, 
teachers' support, etc. In spite of all 
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the limitations, salient studies to dem- 
onstrate the impact should be carried 
out to promote successful educational 
implementation. 

Currently, there are asignificant number 
of initiatives assessing and monitoring 
the quality of ICT use and its impact 
on education. SITES (the second 
information technology in educational 
study), sponsored by the International 
Association for the Evaluation of 
Educational Achievement (IEA), is an 
exemplary study, which identifies and 
describes the educational use of ICT 
across 26 countries in the world. The 
study collected data from different 
stakeholders, and compared and inter- 
preted the results based on the rela- 
tionships of various factors affecting 
the educational use of ICT (Pelgrum 
and Anderson, 1999; Kozma, 2003). 
The OECD has also emphasised the 
need for clarifying the effects of ICT 
use comparing PISA results. European 
Schoolnet published a technical report 
to provide comprehensive information 
on the impact of digital technologies 
on learning and teaching using inter- 
national evidence (Balanskat, Blamire 
and Kefala, 2006). 

In the meantime, the Korean Ministry 
of Education, Science and Technology 
(MEST) has the opportunity to work 
on the impact studies of ICT use on 
educational performance in coopera- 
tion with the OECD. For better under- 
standing of the relationships of ICT 
use and educational performance, 
this paper will provide a theoretical 
mapping of various factors affecting 
ICT use in education by using a con- 
ceptual framework, which was a part 
of the findings of the Korean study, 
and a summary of key findings of a 
nationwide investigation conducted 
in Korea. Constructing a conceptual 
framework is a useful way to connect 
all aspects in a study, and then it may 


guide further investigations into the 
implications of the findings. 

II. ICT use in school 
settings 

School experience was formerly a 
critical resource for humans to obtain 
knowledge and skills in their lives, but 
other sources and methods are now 
available to access new information 
and to interact with people in today's 
knowledge-based society. Computer 
and Internet technologies will probably 
open a door that will make human life 
different. While, in the late 20th cen- 
tury, students asked questions to their 
teachers when they had a question in 
a perplexing situation, children in the 
21st century might choose access to 
the Internet first and use information 
search engines like Google for solv- 
ing their questions and problems. 
However, school is still an essential 
environment for individuals' experi- 
ences on the road to success and to 
promote better adjustment in a soci- 
ety. That is why many educational 
practitioners and policymakers pay 
attention to utilising ICT for improving 
education in school settings. 

There are three major uses of ICT 
in school education (Taylor, 1980; 
Smaldino, Lowther and Russell, 2008; 
White, 1997), as follows. 

First, ICT is used to improve teaching 
and learning — this includes the use 
of application software as a teaching 
and learning tool. Teachers can use 
ICT for presenting information to their 
learners, for assessing and monitor- 
ing learners’ achievements and for 
their own professional development. 
Learners may use ICT for getting 
access to new information, augment- 
ing existing knowledge, sharing what 
they have learned with others, work- 
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ing on school projects with peers and 
acquiring new knowledge and skills. 

The second use is to enhance admin- 
istrative productivity — such adminis- 
trative services as grading and keep- 
ing records in schools are vital for 
tracing a student's learning history 
and monitoring each student’s per- 
formance. The automated administra- 
tive services using ICT are beneficial 
to all stakeholders in schools. 

Third, ICT is used to build information lit- 
eracy — the school curriculum includes 
ICT as a learning object for students. 
The ultimate goal of ICT education 
is to develop ICT skills for problem- 
solving in real life. The main contents 
may include computer architecture and 
cyber ethics. ICT is an indispensable 
tool for people living in this society. 
Teachers who have ICT skills can effec- 
tively prepare teaching materials using 
computers and present complex ideas 
better than those who have fewer ICT 
skills. Students who have ICT skills can 
also be successful in their learning and 
achieve greater outcomes than others 
who have fewer ICT skills. 

The irreversible influence of ICT will 
eventually revolutionise the way we 
learn and teach but the revolution 
may be not remarkable viewed over a 
short time. In particular, the changes 
in educational settings are very slow. 
It is also hard to determine the posi- 
tive influence of ICT use in educational 
performance in schools, because 
assessing the impact is complex, and 
lots of factors affect the processes and 
outcomes of ICT use (White, 1997). 
Educational performance in school 
settings can be interpreted in various 
ways. From the perspective of learn- 
ers, educational performance may 
refer to learning achievement and out- 
comes obtained from the prescribed 
learning contents and activities. These 


include the mastery of content knowl- 
edge, basic skills and attitudes as well 
as core competencies needed in this 
modern society. On the teachers' side, 
educational performance might refer 
to teaching competencies, pedagogi- 
cal content knowledge and teachers’ 
roles in the learning processes and 
outcomes. For educational administra- 
tors, educational performance relates 
to drop-out rate, underachievement in 
school work, entrance rates to higher 
education, reputation ratings from 
stakeholders outside of schools and 
so forth. The learner’s performance, in 
most cases, will be a key component 
to assess educational performance in 
school settings. That is why we, first, 
need to clarify the impact of ICT use on 
educational performance in learning 
and from the learners’ point of view. 

III. Conceptual 

framework of ICT 
use and educational 
performance 

Constructing a conceptual framework 
and indicators is a good starting point 
for investigating a complicated phe- 
nomenon, and then providing inte- 
grated perspectives, even though the 
process has some limitations (Kikis, 
Scheuerman and Villalba, 2009). In 
this paper, a conceptual framework 
indicates various factors that pro- 
foundly influence both ICT use and 
the educational performance of learn- 
ers. This framework was generated as 
a result of comprehensive literature 
reviews and expert reviews. As shown 
in Figure 1, the factors are classified 
into three levels surrounding ICT use 
and educational performance: the 
classroom setting (micro level), the 
school and local community (meso 
level) and regional and national enti- 
ties (macro level). 
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[ Macro level ] 
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Teacher practice 
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Teacher (Ed. background/innovation history/experience with tech. /norms) 
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School types and location/school organisation/local culture/intended curriculum 
staff development/ICT infrastructure/technical support/innovation history 


Economic forces/cultural norms/ed. goals and problems/ed. funding/ 
curriculum standards/teacher standards/ICT policies/ICT infrastructure 
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Figure 1: The conceptual framework of ICT use and educational performance 


ICT use and its impact on educational 
performance may be influenced by 
various factors such as the personal 
attributes of teachers and students, 
and curriculum and teaching practices 
at the micro level. Atthe meso level, the 
school environment and its surround- 
ing factors may affect the use of ICT 
in educational practice. At the macro 
level, ICT use and educational per- 
formance may be influenced by socio- 
cultural norms, economic forces and 
technological advances. This paper 
focuses on understanding the effect of 
ICT use on educational performance 
at the micro level and controls meso 
and macro level variables as con- 
stants either by random selection or 
by setting research boundaries. 

ICT use 

ICT is characterised as a networked 
computer that can process and com- 
municate information in this study. 
However, stand-alone computers and 


portable devices, such as cellular 
phones, are included in ICT use as 
well. Individuals may use ICT in their 
daily lives, and their use may have a 
considerable influence on personal 
performance. The following three 
dimensions are employed to clarify the 
patterns and frequency of ICT use. 

Places in ICT use 

Place in ICT use is divided into two cat- 
egories, in-school and out-of-school, 
based on the location where learners 
use ICT. Most education in schools 
focuses intensively on preparing stu- 
dents to acquire academic skills and 
life competencies. On the other hand, 
interest has recently been growing in 
ICT use for informal learning outside 
of schools and the idea that students 
can benefit from the extracurricular 
use of ICT. There is no doubt that 
individuals spend much time using 
ICT in daily life. They make use of 
ICT for finding information, shopping 
for commercial goods online, chatting 
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Figure 2: The dimensions of ICT use 


with others and playing online games. 
Along with these digital lifestyles, ICT 
use by children and youths might have 
some influence on their thinking and 
learning styles in schools. 

Purposes of ICT use 

The category of purposes of ICT use 
indicates a set of classifications for the 
reason of ICT use and the intentional- 
ly of learning, which include learning 
and entertainment. Learners may use 
ICT for their learning needs, such as 
obtaining knowledge, solving complex 
problems and acquiring new skills. 
Experiences that learners have with- 
out any specific intention of learning 
may be categorised as entertainment. 
ICT also creates new entertainment 
environments in which learners can 
socialise with friends and play games. 

Contexts in ICT use 

Learners may work individually or 
socially with peers while using ICT. In 
an individual context, learners use ICT 
alone without collaborating with others. 
For instance, a student involved in 


a project for solving a problem can 
use software to present ideas and 
thoughts. A social context refers to a 
setting in which two or more learners 
use one computer together, or in which 
a learner works with friends to perform 
collaborative tasks online. Such tools 
as wikis, blogs and bulletin boards 
might be used by learners to interact 
with others. For example, students 
could use a wiki for the collaborative 
development of a project. 

Educational performance 

The meaning of educational perform- 
ance is vague and diverse depending 
on domains, despite the long history of 
research and attention from academia 
as well as practitioners. Based upon 
previous studies, educational per- 
formance may be conceptualised as 
a futuristic concept that encompasses 
not only the traditional concept of edu- 
cation but also the extended version 
of human learning. The educational 
performance of learners is defined as 
the processes and results of perform- 
ance, which are revealed internally 


193 


Chapter IV — Case studies 


and externally through the integration 
of essential knowledge, skills and atti- 
tudes, and the continuing construction 
of experiences with ICT use. 

To make operational definitions of 
complex educational performance, the 
study suggests a two-dimensional tax- 
onomy model, which is composed of 
six cells within the two dimensions: (1) 
three performance domains (cognitive, 
affective and sociocultural) by (2) two 
behaviour levels (internal, external). 
This model utilises the approaches 
of Bloom’s taxonomy of educational 
objectives and Krathwohl’s taxonomy 
of affective competencies. It also 
puts more emphasis on socio cultural 
aspects and less on psycho motor 
aspects than other approaches. 

The performance dimension contains 
three categories: cognitive, affective 
and socio cultural. These three cat- 
egories are assumed to be mutually 
independent and, at the same time, 
to be critical for learners in the future. 


Traditional educational taxonomies 
emphasised cognitive categories with 
less, if any, emphasis on the affective 
and sociocultural dimensions. As the 
world evolves into a more post-mod- 
ern society, however, where multiple 
voices are heard, its citizens, includ- 
ing the younger generation, should be 
sensitive to socio cultural performance. 
The presence of ubiquitous computing 
technology connected in a global net- 
work will also accelerate sociocultural 
dynamism. 

These categories are assumed to lie 
along a continuum from internalised 
(or centripetal) behaviour, to external- 
ised (or centrifugal) behaviour. The 
continuum underlying the behaviour 
levels is assumed to be the orientation 
of performance; that is, internal com- 
petencies are believed to be oriented 
more toward the learners themselves, 
while external competencies relate 
more to the world and others outside. 
In the new millennium, learners are 
expected to be more participatory and 
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competencies 


External 

competencies 
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Figure 3: Conceptualisation of educational performance in learners' perspectives 
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active practitioners who will contribute 
to the betterment of the community 
and the world. To live as active prac- 
titioners, learners should understand 
the cognitive, affective and sociocul- 
tural aspects of the world to make it a 
more liveable place. Recent epistemo- 
logical perspectives such as those of 
Leontev’s activity theory and Lave and 
Wenger’s situated cognition theory 
also confirm this internal-to-external 
developmental orientation. 

The following descriptions briefly 
explain the six cells constructed by 
three performance domains and two 
behavioural levels. 

Cognitive-internal competency: This 
refers to the individual’s internal abil- 
ity to select and gather information, 
and construct knowledge. 
Cognitive-external competency: Cogni- 
tive-internal competency is mani- 
fested as useful tools for transform- 
ing the individual’s situated life- 
world. Effective problem solving is a 
relevant example. 

Affective-internal competency: To 
live as an independent and mature 
member of many overlapping com- 
munities, a learner should have a 
set of internal values to recognise 
the importance of oneself as well 
as of others. Individuals should also 
be able to appreciate social norms 
such as the importance of honesty 
and integrity. 

Affective-external competency: Mature 
individuals are those who act in 
accordance with their own true 
values in adverse as well as favour- 
able situations. Self-efficacy, goal- 
setting and perseverance are a few 
examples. 

Sociocultural-internal competency: 
As future societies will be more 
socially diverse, individuals need 
to tolerate and appreciate one 
another. This sociocultural perform- 


ance begins with open-mindedness 
toward uncertainty. Members should 
also be equipped with global com- 
munication skills such as foreign 
language proficiency and cross-cul- 
tural understanding. 
Sociocultural-external competency: 
If one fully recognises the presence 
of others and acquires communica- 
tion skills, then one may be ready to 
collaborate with others to make the 
community a better one. Assuming 
proactive roles, such as those of 
leadership, performing social serv- 
ices and maintaining strong ties with 
others in a community are some 
exemplary behaviours. 

IV. Findings from 
a nationwide 
investigation in Korea 

Based upon the conceptual framework 
presented briefly in the previous chap- 
ters, a nationwide investigation was 
conducted in Korea. Measurement 
scales were developed for measuring 
the types of ICT use and educational 
performance of learners in the investi- 
gation. The scales consisted of 42 test 
items in ICT use and 33 items in edu- 
cational performance using the self- 
reporting method with a 4-point Likert 
scale. Expert reviews and pilot tests 
were conducted for validation. During 
the expert review, the experts evalu- 
ated and provided comments on both 
conceptual frameworks and measure- 
ment scales quantitatively and quali- 
tatively. Item analyses, reliability tests 
and validity tests were employed to 
optimise the measurement tools for 
ICT use and educational performance 
through three pilot tests. The nation- 
wide investigation was conducted 
among 1 071 first-year high school stu- 
dents (15-year-olds) during the three 
weeks at the end of a school term. 
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The overall interpretation of the results 
in the investigation indicates that ICT 
use and educational performance 
were significantly connected. ICT 
use has a positive influence not only 
on cognitive competencies enhanced 
through traditional education systems, 
but also on affective and sociocultural 
competencies required for individuals 
in future societies. The findings from 
the investigation are summarised as 
follows (Kang, Heo, Jo, Shin, Seo and 
Shin, 2008). 

First, using ICT outside school influ- 
ences an individual’s educational 
performance more than using it in 
school settings. In most cases, learn- 
ers can get access more conveniently 
in homes and commercial computer 
rooms outside schools than in schools. 
Schools still provide limited access 
to learners. Teachers, probably, are 
responsible for ICT use in both class 
hours and after classes. Most activi- 
ties using ICT in class hours are to 
present information to students by 
teachers. Few opportunities may be 
provided to students to use computers 
except for special needs. It suggests 
that we should rethink how to use 
ICT in schools and integrate learners’ 
experiences in informal settings into 
school learning. 

Second, when individuals use ICT for 
their learning rather than entertain- 
ment, it generates a positive impact on 
educational performance. It means that 
such activities as playing games and 
listening to music may not enhance 
educational performance much, even 
though some studies on the educa- 
tional use of games reported its posi- 
tive impact on learning outcomes. As 
an investigation on types of games that 
most individuals use for their enter- 
tainment reports, violent games may 
spread more than othertypes of games 


(Ferguson, 2007). However, participa- 
tion in online communities as an activ- 
ity outside schools positively affected 
sociocultural competencies rather than 
the other two competencies. 

Third, ICT use in individual contexts 
resulted in a more positive influence 
on learners’ educational perform- 
ance than using it in social contexts. 
When individuals use ICT for their 
learning outside schools, it possibly 
enhances the cognitive, affective and 
sociocultural competencies of their 
educational performance. Using ICT 
in social contexts also has a small 
positive impact on their educational 
performance. Collaborative learning 
outside of schools as a learning activ- 
ity in a social context may enhance 
educational performance. It indicates 
that collaborative learning while solv- 
ing real problems, and working on 
authentic projects must be included 
for better learning. Web 2.0 tools, one 
of the recent technologies, have been 
widely used in many situations and are 
expected to provide more opportuni- 
ties for sharing ideas and cooperating 
among individuals in social contexts. 

It is evident that ICT use affects learn- 
ers’ educational performance posi- 
tively, but its impact is mainly on the 
cognitive development within their 
educational performance. This study 
assumed that the integration of cogni- 
tive, affective and sociocultural com- 
petencies is important for individuals 
to be successful in current and future 
society. Even though ICT use did not 
influence affective and sociocultural 
competencies much, more attention 
should be paid to possible methods for 
using ICT for developing those compe- 
tencies in and out of school settings. 
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V. Conclusion 

The biggest challenge in assessing 
ICT impact on learners’ educational 
performance is to identify the distinctive 
influence of ICT use on it. As mentioned 
earlier, educational performance is a 
vague concept and difficult to define 
and to measure, and various attributes 
of learners and complicated features of 
the external environment surrounding 
learners might affect their performance 
in the present and future. However, it is 
not a matter of whether the impact of ICT 
use can be exactly measured. We need 
to pay more attention to what and how 
to measure, and to make interpretations 
to promote better performance. The 
following aspects should be taken 
into account in possible further impact 
studies on ICT use in education. 

First, more interest needs to be taken 
in making connections and studying 
the relationships among various fac- 
tors that influence ICT use in educa- 
tion. This paper elucidates diverse 
factors on three levels, referring to 
school settings and the related supra 
systems. Among them, some factors 
relate directly to learners' performance 
and some others indirectly. It will lead 
to the construction of another frame- 
work for comprehensive interpretations 
and future development. 

Second, ICT use in informal learning 
must be examined for a better under- 
standing of ICT use in learners’ perform- 


ance. Sometimes, individuals use ICT 
in personal contexts (home, cafes and 
pupils’ houses) more than in schools 
and then those experiences can affect 
ICT use in schools in some ways. 

Most activities in a school setting 
might be predetermined by teachers 
and through national standards, but all 
experiences and activities outside of 
schools cannot be estimated precisely. 
When another new world where cyber- 
space and physical space are com- 
bined in one space opens, the apparent 
distinction between formal and informal 
learning may disappear. Since this is 
the case, ICT use in informal learning 
that happens to learners unintention- 
ally should be paid more attention by 
educational practitioners. 

Third, the quantitative and qualitative 
approaches in assessing and inter- 
preting the impact of ICT use in edu- 
cation should be combined for the 
comprehensive understanding of this 
emerging phenomenon. While the 
quantitative approach answers best 
to problems requiring a description of 
trends or an explanation of the relation- 
ships among variables, the qualitative 
approach will address questions refer- 
ring to the exploration of little-known 
situations or a detailed understanding 
of a central phenomenon (Creswell, 
2008). Unknown factors affecting ICT 
use in education may be found through 
qualitative methods of evaluation. 
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Introduction 

The relationship between information 
and communication technologies (ICT) 
and improved teaching and learn- 
ing has increasingly been the focus 
of interest for education policymak- 
ers, researchers and other education 
stakeholders after two decades of ICT 
investment and integration in schools 
across Europe. What impact or dif- 
ference can ICT make in education 
systems? How can ICT be a motor for 
improvement, progress, educational 
change and innovation? The inter- 
relationship between policy, practice 
and research has likewise become 
an important focus within the area of 
evidence-based policymaking. 

The ICT impact report — a review of 
studies on the impact of ICT in educa- 
tion produced by European Schoolnet 
in the framework of the European 
Commission's ICT cluster — revealed 
considerable gaps in what is known at 
a European level about the impact of 
ICT in schools. 

Evidence or access to evidence on the 
impact of ICT in schools is unevenly 
spread across Europe. Many of the find- 
ings relate to the United Kingdom and to 
England in particular. They are mostly in 


(’) This paper draws on longer studies in STEPS 
written by the author, A. Balanskat, T. Husing, 
W. Korte, B. van Oel and L. Sali. 


English. There are gaps in what is known 
about other countries. No doubt some evi- 
dence exists and efforts should be made 
to identify it and ensure it is translated. If 
it does not exist, efforts should be made 
to support transnational studies to ensure 
good coverage and reliable results. 
(Balanskat, Blamire and Kefala, 2006) 

The ‘Study of the impact of technology 
in primary schools’ (STEPS) sought to 
close this gap and to provide a more 
balanced and comprehensive picture 
of the impact of ICT on primary educa- 
tion. The study was commissioned by 
the European Commission Directorate- 
General for Education and Culture ( 2 ) 
and undertaken jointly by European 
Schoolnet (EUN) and Empirica GmbH 
between January 2008 and June 
2009. Empirica was responsible for 
the Learnlnd survey of 30 000 teach- 
ers and head teachers in 27 European 
countries for the Directorate-General 
for the Information Society and Media 
(Empirica, 2006): this provided quanti- 
tative evidence on the access and use 
of ICT in European schools in 2006 
generally in primary and secondary 
education. Based on the experience of 
both organisations in the field and the 
application of different approaches and 
methods (quantitative and qualitative) 


( 2 ) This study was financed (at a cost of 
EUR 232545) by the European Commission. 
Contract EACEA-2007-3278. Opinions presented 
in this chapter do not reflect or engage the 
Community. © European Commission. 
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for gathering and analysing develop- 
ments in ICT in education, European 
Schoolnet and Empirica worked in 
a complementary way to paint a rich 
portrait of the impact of ICT on primary 
education. 

The main purpose of STEPS was to 
produce a comparative analysis of the 
main strategies for the integration of 
ICT in primary schools in the EU-27, 
Iceland, Liechtenstein and Norway, 
their impact and future development 
perspectives. The study aimed to iden- 
tify the impact of ICT at three levels: on 
learning and learners, on teachers and 
teaching and on primary school devel- 
opment plans and strategies. It sought 
to identify the main drivers and ena- 
blers for effective and efficient use of 
ICT, and to propose recommendations 
on the integration of ICT in education 
for policymakers and stakeholders. 

The challenge was considerable: to 
identify commonalities across 209 866 
schools ( 3 ) offering primary-level edu- 
cation in the 30 countries surveyed, 
ranging from 14 in Liechtenstein to 
55 329 in France. Moreover, compul- 
sory schooling in the countries cov- 
ered begins between the ages of four 
and seven and most primary schools 
are managed, funded and governed 
by the local municipal councils and so 
data tend to be held locally and are 
not always available. 

The final report amounted to some 
66 separate reports totalling over 
1 000 pages. In the following sections, 
the approach and main findings are 
outlined. 


( 3 ) A primary school is defined as one that 
educates children between the ages of four 
and 1 1 . The figures do not include private schools 
or kindergartens. A number of countries have 
all-age schools or combine primary and lower 
secondary schools in one school. 


Approach 

The methodological challenge was 
considerable. Strategy and impact 
were the two underlying concepts of 
the STEPS study. They can be seen 
as the two ends of a chain: a strat- 
egy is always designed with the aim 
of having impact. Strategies and poli- 
cies are shaped at several levels, and 
this makes policy implementation and 
evaluation a difficult task, especially 
because they involve attitudinal and 
work process changes. How do we 
know whether it was the interven- 
tion that made the impact without 
taking other factors into account? Can 
change attributed to an ICT strategy be 
isolated from other factors? How was 
policy implemented in practice? How 
do we measure impact? As Gordezky 
et al. note: 

Changing a large complex school system 
is a messy business. Results from change 
efforts are often unpredictable, show up 
in ways that are difficult to quantify, and 
can lead to counterintuitive and undesir- 
able consequences. (Gordezky, Marten 
and Rowan, 2004) 

A number of strategic layers play a 
role when looking at the implementa- 
tion of ICT. Strategies can be found 
from societal level all the way down to 
an individual teacher making strategic 
decisions on when and how to use ICT. 
These levels include, first, society at 
large and how it tackles ICT; second, 
the education system (including policy 
targets and the main actors). The third 
and fourth layers are formed by gov- 
erning bodies (e.g. regional or local 
authorities) and by individual schools. 
A final layer is the ‘end-user’: often 
the teachers, but also the learners 
themselves. These end-users develop 
strategies to comply with national, 
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regional and local requirements; and 
of course to satisfy their own targets. 

Impact can be described as the over- 
all achievement of an intervention on 
these domains within the educational 
system and can be described by a 
variety of qualitative and quantitative 
indicators such as ‘improvements in 
national tests’ or ‘improved learning in 
schools’depending on the policy target. 
It is the end point of an intervention 
involving input, process, output and 
outcome. Isolating the variable which 
actually causes the impact is problem- 
atic in education. Within STEPS, the 
following definition of impact was used: 
‘a significant influence or effect of ICT 
on the measured or perceived quality 
of (parts of) education’. The study was 
based on the assumption that not all 
impacts are positive or intended, that 
not all policies are implemented as 
planned and that classroom practices 
are hard to change (see McLaughlin, 
2005). Although evidence about effec- 
tive strategies has been identified, 
policies are generally shaped to local 
contexts and practices take a long 
time to change. Years of ICT impact 
studies confirm this complex picture. 
ICT impacts cannot always be meas- 
ured through test scores — sometimes 
no gain in test scores can be found 
and no direct link can be established 
between an ICT intervention and 
improved attainment. One solution in 
this study was to look at impact not 
only in attainment (hence the broad 
definition of impact) but also to look at 
how ICT improves processes of teach- 
ing and learning within the school. 

A multi-perspective approach was 
adopted for STEPS, taking into 
account evidence from stakeholders 
(policymakers, teachers and head 
teachers), research and site visits 
to schools (including interviews with 


learners). Evidence came from five 
sources: 

a policymaker survey in the 30 
countries to provide an overview of 
policy approaches to ICT in primary 
education; 

an analysis of quantitative data from 
over 18 000 teachers and head 
teachers interviewed for the 2006 
Learnlnd ICT benchmarking survey 
(Korte and Husing, 2006); 
a review and analysis of the evidence 
from over 60 research studies pub- 
lished in more than 20 countries; 
250 responses to a school survey 
seeking qualitative insight into the 
impact of national strategies in 
schools, and the identification of 
good practices via self-reporting; 

25 case studies documenting the 
good practices identified. 

Policy survey 

The policy survey was the main tool 
for deepening knowledge of national 
and regional strategies and was in 
three parts: 

general information about the char- 
acteristics of the primary school 
system (ranging from the number of 
schools, curriculum, teachers’ pay 
and conditions to school govern- 
ance) and emerging policy trends 
and priorities; 

the use of ICT in primary schools, 
covering ICT resourcing, teacher 
skills development and ICT support, 
the place of ICT in teaching and 
learning; 

ICT policy for primary schools, 
including ICT in education policy, 
examples of strategies and good 
practice. 

The policy survey was completed 
between July 2008 and March 2009. 
National correspondents (in most 
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cases nominated by ministries of 
education) gathered information on 
national or regional policy contexts, 
often translating documents only 
available in the local language. This 
was supplemented by information 
from other STEPS sources (Learnlnd 
data, school surveys and the lit- 
erature review) and by data in the 
public domain (EUN insight country 
reports ( 4 ), Eurydice ( 5 )). 

The results of the policymaker survey 
were analysed and presented in 
Report 1, ‘Policy survey results and 
analysis’, providing an overview and 
comparison of policies and types of 
strategies. Summaries of national pol- 
icies were also included in 30 country 
briefs. 

Teacher survey 

Quantitative data in the Learnlnd sur- 
veys used standardised interviews with 
head teachers and class teachers (a 
random sample) in 27 European coun- 
tries collected in 2006. The sample 
was split between primary, lower sec- 
ondary and upper secondary schools, 
but STEPS concentrated on the results 
of primary schools only. In total, 12 379 
interviews with classroom teachers and 
6 449 interviews with head teachers of 
schools which offer primary education 
were carried out. 

The use of ICT in European primary 
schools was measured using the fol- 
lowing criteria: 

teachers’ attitudes and motivation 
with regard to ICT, including per- 
ceived impact of ICT; 


( 4 ) http://insight.eun.org/ww/en/pub/insight/misc/ 
country_report.cfm 

( 5 ) http://eacea.ec.europa.eu/portal/page/portal/ 
Eurydice/Overview/OverviewByCountry 


technical infrastructure in schools, 
including computer equipment and 
Internet connectivity; 
the use of ICT In class and for edu- 
cational purposes; 

ICT competence of teachers; 
barriers to ICT use as perceived by 
teachers and head teachers. 

The results of the Learnlnd data anal- 
ysis were presented and discussed in 
Report 2, ‘Learnlnd data results and 
analysis’. Summaries of the data anal- 
ysis per country can be found in the 30 
country briefs. 

Literature review 

The main scope of the literature review 
was qualitative rather than quantita- 
tive, in order to ensure sufficient cov- 
erage from participating countries. The 
aim was to identify and summarise in 
English recent studies (up to four per 
country) that gave important insights 
in the field and to include countries 
where information access has so far 
proven to be difficult due to language 
barriers and fragmented research as 
revealed by the ICT impact report. 

The appointment of committed key 
experts from existing partner networks, 
from a wide geographical area (north, 
south, east and western Europe) and 
especially in those countries where 
until now information had been unob- 
tainable, enabled important studies in 
those countries to be identified, and, 
most importantly, to make the results 
of these studies more widely known. 
Summaries of research in each country 
were presented in the country briefs. 

School survey 

The STEPS school survey aimed to 
gather examples of the integration of 
ICT in primary school daily activities 
and to obtain a snapshot of current 
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views of teachers on ICT use and 
impact in their school. The survey 
consisted of an online questionnaire 
with both closed and open questions 
in nine languages. 

Case studies 

The purpose of the case studies was 
to find out more about effective use 
of ICT and enablers or barriers at dif- 
ferent levels of the education system. 
The case studies sought to show how 
the strategies of policymakers, schools 
and teachers impacted on teaching 
and learning. 

The case studies were designed to 
show the richness of implementation 
and also to describe a number of typ- 
ical situations in sometimes quite dif- 
ferent schools and contexts. In most 
cases, the visits were related to a 
specific application of ICT or a project 
which had been identified by ministries 
of education or schools themselves as 
demonstrating good practice. 

Within STEPS, the case studies 
helped to: 

• visualise what happened in the 
classrooms: 


include the voice of teachers, pupils 
and school leaders; 
complement the evidence base 
by an in-depth investigation and 
observation. 

In total, 25 contrasting schools in 
13 countries were selected for a case 
study visit. The case study (written by 
an evaluation team) followed a fixed 
format. At school, teacher and learner 
levels, the reporters were asked to 
highlight impact, enablers and bar- 
riers. All the case studies were ana- 
lysed in terms of themes, issues and 
typologies and presented in Report 5, 
‘Case study analysis’. 

Key findings 

An analytical framework was devel- 
oped early in the project and used for 
the integrated analysis and presenta- 
tion of the overall findings. The frame- 
work visually captures key elements 
and represents them in a logical and 
concise way. The analytical frame- 
work is built around a core of teachers, 
learners and the school as a whole. 
The framework helps to describe the 
context in which ICT is introduced and 
implemented. 
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The model consists of five levels: soci- 
ety, education system, school, teach- 
ers and learners. These levels repre- 
sent where strategies, enablers and 
barriers can be found. The framework 
reads from left to right, representing 
not only a hierarchical flow but also a 
flow from strategy to impact. 

Asynthesis report was compiled taking 
into account the results of the five con- 
tributory reports described above. It 
presented key findings, conclusions 
and recommendations for future work. 
The key findings are summarised 
below, together with suggestions for 
furtherinvestigation. They are grouped 
under four headings: impact on learn- 
ers and learning, impact on teachers 
and teaching, impact on schools and 
planning and system-wide findings. 

Impact on learners and learning 

ICT improves children’s 
knowledge, skills and 
competences 

There is a broad consensus among 
primary teachers about the positive 
impact of ICT on learners and learn- 
ing. Research shows that a range of 
skills and competencies are acquired 
by the use of ICT: digital, communi- 
cation, language (first and second), 
social and cognitive skills. Teachers 
interviewed in the Learnlnd survey 
note a positive impact on basic skill 
acquisition (reading, writing and arith- 
metic) through the use of ICT — and 
research echoes this finding. UK 
research shows that English, maths 
and science test scores improve with 
ICT, and a Hungarian study shows that 
ICT-rich constructivist learning envi- 
ronments improve learning outcomes, 
especially for disadvantaged children. 
Many case studies highlight how ICT 
helps children understand the subject 
they are studying and caters for indi- 


vidual needs, although schools find it 
hard to isolate the contribution of ICT 
to test scores. 

However, research suggests thatthere 
is a discrepancy between children's 
under-use of ICT at school and their 
more frequent and often more sophis- 
ticated use at home. Although a range 
of digital skills are acquired outside 
school informally, some basic compu- 
ter skills are not. 

ICT increases motivation, 
confidence and engagement 
in learning 

Some 87 % of teachers say that pupils 
are more motivated and attentive 
with ICT — according to the Learnlnd 
data. Much of the research suggests 
that ICT has a positive impact on stu- 
dent attendance, behaviour, motiva- 
tion, attitudes and engagement, that 
guided, active and enquiry-based 
tasks with ICT are particularly motivat- 
ing, and that technology enables finer 
differentiation and personalisation. A 
large-scale comparative study shows 
that pupils participate more actively in 
learning when ICT is used. Teachers 
in the school survey felt strongly that 
ICT is a means of overcoming low 
motivation, social diversity and dis- 
engagement. In the case studies, 
there are examples of schools using 
ICT to improve links between learning 
inside and outside school and involve 
parents. ICT also impacted on group 
processes and collaborative learning. 

Assessment can be 
more sophisticated and 
individualised 

ICT-based assessment systems used 
in some case study schools give more 
sophisticated feedback to teachers, 
parents and pupils on their perform- 
ance, e.g. through the analysis of 
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Figure 2: United Kingdom: motivated confident and engaged independent learners 


test scores. Virtual learning environ- 
ments enable the individual track- 
ing of progress and help identify 
the next ‘learning step’, so enabling 
pupils themselves to detect errors and 
shortcomings. Achievement can be 
recorded in e-portfolios. 

Areas for further investigation 

Longer-term study of the impact of 
ICT on improving learning achieve- 
ment, also taking into account the 
effects of differing learning styles. 

• How quality assurance and inspec- 
tion regimes are developing to take 
full account of ICT developments. 
Subject-specific investigation of ICT 
impact and potential, especially in: 
— key priority subjects, in particu- 
lar mathematics, science and 
technology; 


— subjects where resource devel- 
opment by individual teachers is 
difficult and/or costly. 

Almost all aspects of assessment: 
developing effective tools to meas- 
ure ICT skills; enabling ICT deploy- 
ment by students within the assess- 
ment process; e-assessment; etc. 
Development of indicators on suc- 
cessful use of ICT in relation to dif- 
fering learning tasks and contexts. 
Understanding the feasibility, costs 
and benefits of personalised learn- 
ing. 

Impact on teachers and teaching 

Most teachers use ICT and are 
‘ICT-optimistic’ 

Some 75 % of primary teachers (and 
their pupils) use computers in class 
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Figure 3: Spain: after-school on-site training, responsive to needs, with a pedagogical expert on hand 


according to the Learnlnd data: from 
around 90 % in the Nordic countries to 
approximately 35 % in Greece, Latvia 
and Hungary. Teachers find that ICT 
supports in equal measure a range of 
learning and teaching styles, whether 
didactic or constructivist, in passive 
activities (exercises, practice) and 
in more active learning (self-directed 
learning, collaborative work). The 
research shows that rich constructivist 
learning environments improve learn- 
ing outcomes, especially for learners 
from disadvantaged areas. Teachers in 
some countries (the United Kingdom, 
Cyprus, the Netherlands, Portugal 
and Poland) are more optimistic about 
ICT than others (Sweden, France and 
Austria). Nevertheless, a significant 
minority (21 %) consider that using 
computers in class does not in itself 
have significant learning benefits. 


There is little to no correlation between 
impact-optimism and levels of school 
equipment, sophistication of use or 
even teacher skills. There is a cluster 
of countries with high skill levels and 
high expectations as to ICT impact: 
the United Kingdom, the Netherlands, 
Cyprus and Malta. 

ICT is pedagogically 
under-used 

Despite the high levels of reported 
classroom use mentioned above, 
according to some studies teach- 
ers use ICT more for administration, 
organisation and planning. They also 
indicate that teachers are aware of the 
potential benefits of ICT for students, 
have a positive perception of ICT in 
terms of supporting active autono- 
mous learning and creating authentic 
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tasks, but lack the pedagogical vision 
to integrate ICT effectively in teaching. 
The research shows that ICT can pro- 
mote new pedagogical approaches, 
but only if ICT is fully integrated into 
subject lessons. In the Nordic coun- 
tries, teachers in primary schools 
more often regard ICT as supporting 
their pedagogy than teachers in sec- 
ondary schools. 

Quality training increases 
teachers’ motivation and digital 
and pedagogical skills 

Teachers responding to the good prac- 
tice survey consider that using ICT 
improves their motivation and teaching 
skills. We know from the policy survey 
that the 30 countries are investing in 
developing teacher ICT skills; but that 
in a significant number of countries 
teachers entering the profession may 
have little formal training in using ICT 
in teaching. Researchers have drawn 
some worrying conclusions about the 
effectiveness of continuing professional 
development in ICT: that teachers have 
failed to acquire the desired level of 
ICT skills for classroom instruction and 
that training has not translated into 
gains in pupil learning. Research sug- 
gests that teachers adapt more easily 
to new technologies through a step-by- 
step approach with minimal disruption, 
and that on-site is preferable to off- 
site training. Training courses failed to 
match needs and lack the pedagogical 
and practical dimension, according to 
the analysis of responses to the policy 
survey. The survey also indicates that 
reliable technical back-up and inspiring 
pedagogical support for teachers are 
often missing. 

Areas for further investigation 

Pinpointing sound pedagogy and 

understanding whether and in what 


ways ICT specifically can enhance 
teaching and learning. 

Developing fully integrated models 
of ICT-supported learning delivery 
which provide examples and tem- 
plates to guide local development. 
The environment and conditions for 
continuing professional develop- 
ment for teachers in relation to ICT. 
Improving interoperability in the 
interests of maximum exchange, 
deployment and sharing of teaching 
materials. 

Impact on schools and ICT 
planning 

Children’s access to 
technology is improving 

Analysis of the 2006 Learnlnd data 
reveals that almost all primary schools 
use computers, with at least 88 % 
of schools in each country having 
Internet access and with an average 
of eight Internet computers per 100 
pupils. However, there are huge vari- 
ations in ICT infrastructure and con- 
nectivity across and within countries. 
The computer-to-pupil ratio ranges 
from Luxembourg (23 computers per 
100 pupils), Denmark and Norway 
(18), the United Kingdom (16) and the 
Netherlands (15) to much lower fig- 
ures for Latvia, Lithuania and Poland 
(6) and Greece and Slovakia (5). 

According to figures provided for the 
policy report, the computer-to-pupil 
ratio now ranges from 3.1 to 32 per 
100 pupils and eight countries have 
more than 14 computers per 100 pupils 
(representing over 50 000 schools). 
Some 72 % of the study’s 209 866 pri- 
mary schools have broadband and in 
20 countries over two thirds of primary 
schools have broadband. Interactive 
whiteboard provision ranges from very 
few (e.g. Finland, Norway) to near 
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saturation (the UK, where all primary 
schools have at least one). Denmark, 
Estonia and Norway have the highest 
levels of virtual learning environments 
that offer access from outside school. 
Smaller primary schools are disadvan- 
taged in terms of equipment, accord- 
ing to research, yet case studies show 
that the benefits for schools in small 
communities are considerable. 

Whole school ICT integration 
and leadership matter 

ICT integration in subjects and class- 
rooms is the key to changing teach- 
ing practices, according to research 
— and the school leader’s support 
is crucial in cases where primary 
schools are free to integrate ICT in 
the curriculum. The policy survey sug- 
gests that countries with high levels 
of ICT favour dispersion into class- 
rooms. Some 68 % of primary schools 
have computers in classrooms, rather 
than in computer labs, according to 
the Learnlnd data. This is the case in 
more than 90 % of primary schools 
in Luxembourg, Slovenia, the United 
Kingdom, the Netherlands, Cyprus 
and Ireland. In contrast, there are 
10 countries with computers in class- 
rooms in fewer than 50 % of schools 
(Cyprus, Estonia, Greece, Italy, Latvia, 
Lithuania, Hungary, Poland, Slovakia 
and Spain). In these countries, the 
majority of primary schools use com- 
puters for education in dedicated com- 
puter labs. 

ICT improves administration 
and access to information 

Schools have incorporated ICT into 
management tasks and ICT is increas- 
ingly used by teachers for administra- 
tion and planning. In several case stud- 
ies, school-wide planning improved 
with the help of ICT. This is because 


ICT makes administration accessible to 
wider groups through a web interface 
and school records are more easily 
maintained, exchanged and updated. 
However, research indicates that 
school ICT plans tend to concentrate 
more on infrastructure than on how ICT 
can be used to enhance teaching and 
learning, and this can actually work 
against innovation (as found in some 
case studies). Virtual learning environ- 
ments are becoming more widespread, 
but are used more for administration 
than for learning. Research shows that 
sufficient time is needed to assimilate 
virtual learning environments. However, 
once introduced, they are increasingly 
used by teachers. 

Areas for further investigation 

The economics of ICT investment, at 
both micro (e.g. optimal initial capi- 
tal and human resource investment 
at school level) and macro (e.g. 
the relative effectiveness of local, 
regional and national investment) 
levels. 

Exchange of best practices in reach- 
ing remote and disadvantaged 
communities. 

How ICT in school management can 
support and facilitate the teacher’s 
role and the quality of the educa- 
tional experience. 

Models for managing and supporting 
ICT development and use at school 
level. 

Primary schools systems 

Strategies for ICT tend 
to feature infrastructure and 
teachers’ digital competence 

Responses to the policy survey indi- 
cate that all 30 countries have or 
have recently had at least one ICT 
policy or initiative affecting primary 
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schools, usually aimed at improving 
infrastructure and digital competence 
among teachers; and less frequently 
targeted at the supply of digital learn- 
ing resources, pedagogical reform or 
leadership. From the 74 policies, pro- 
grammes and projects analysed in the 
study, strategies range from a system- 
wide intervention including ICT to spe- 
cific projects focused on, for example, 
equipment, e-safety or teacher educa- 
tor ICT training — and with the locus 
of control running from central govern- 
ment control to high levels of school 
autonomy and responsibility. ICT in 
schools is still a topic that arouses 
controversy; and where the debate 
involves the general public, the con- 
cerns tend to be about e-safety, 
according to the policy surveys. 

Digital competence usually 
features in the curriculum 

Digital competence is in the pri- 
mary school curriculum in 22 of the 
30 countries, according to the policy 
survey, either integrated across sub- 
jects (in 15 countries) or taught as a 
separate subject (in 11 countries). 
Learnlnd data show that teaching ICT 
as a separate subject, computer sci- 
ence, varies across Europe: ranging 
from being taught in nearly all schools 
in Latvia, Poland and Hungary to very 
few in Finland (19 %) and Austria 
(9 %). There is little evidence from the 
Learnlnd data to suggestthat teaching 
computer science as a separate sub- 
ject implies placing less importance 
on ICT in other subjects. There are, 
however, exceptions to this obser- 
vation: in the United Kingdom, ICT 
is used in most subjects in 94 % of 
schools; but at the same time compu- 
ter science is taught separately in only 
52 % of schools. In Latvia, conversely, 
ICT is used in most subjects in 42 % 


of schools and computer science is 
taught separately in 97 %. 

ICT responsibilities within the 
system can be unclear 

In most countries, ICT is part of gen- 
eral education policy and there is also 
a specific ICT policy for all schools, 
but no specific policy for ICT in pri- 
mary schools. In countries where 
ICT has long been used in primary 
schools, policies seem to make fewer 
explicit references to ICT; and so ICT 
could be said to be pervasive and a 
given. Responsibilities can be unclear 
according to the policy survey: while 
primary schools have increasing 
autonomy as public sector services 
become decentralised, ICT responsi- 
bility in the system varies and is some- 
times unclear. Hardware provision is 
often a national or municipal respon- 
sibility, but not maintenance, techni- 
cal or pedagogical support. This can 
leave schools in some confusion. 

Areas for further investigation 

Understanding ways in which 
national and regional strategies can 
address the aim of improving the 
quality of education. 

Whetherthere are differential impacts 
depending on whether ICT skills are 
taught separately or through integra- 
tion in the general curriculum. 
Creating a flow of information on 
future visions for ICT in education 
(e.g. emerging new technologies, 
integration, networking, mix ofschool- 
based and home-based learning). 
Rates of investment in ICT in educa- 
tion: howhave they developed in recent 
years, what are the current trends and 
is investment sustainable? 
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Some early conclusions - 
a personal perspective 

The conclusions and recommenda- 
tions arising from STEPS are still 
under discussion at the time of writ- 
ing. A synoptic report, conclusions 
and recommendations, the five con- 
tributory reports, 30 country reports 
and 25 case studies are to be pub- 
lished online (see http://steps.eun. 
org), together with a paper describing 
the methodology in detail. 

What is certain from the evidence is 
that teachers are at the heart of ICT 
success in Europe's primary schools. 
They are positive about ICT but can 
be frustrated by external (and some 
internal) inhibitors. Teachers need 
ongoing appropriate training and 
quality support driven by pedagogy 
not technology, good digital learn- 
ing resources and room for initiative 
and risk-taking. School leaders and 
municipalities (depending on school 
governance arrangements) would 
benefit from guidance in the use of 
ICT in organisational change and the 
use of tools for whole-school self- 
evaluation. 

Likewise, it is clearthat primary school 
children are excited about technol- 
ogy; they are competent with ICT in 
many (but not all) respects and are at 
home with technology, using it exten- 
sively outside school. This should be 
more actively exploited by schools, 
but sensitively (it is their technology 
and their free time), and ensuring that 
critical gaps like children’s media lit- 
eracy are covered. 

The value and contribution of ICT 
as an enabler for more general edu- 
cational policy visions, reforms and 
objectives could be more explicitly 
stated in policies. The impact of tech- 


nology can then be evaluated in terms 
of its contribution to these wider policy 
aims. Until recently, policy measures 
to encourage the use of ICT have 
tended to focus on improving infra- 
structure and developing teacher 
competence in ICT. From that narrow 
perspective it is more difficult to justify 
the investment. In some recent edu- 
cation policies and initiatives, ICT is 
invisible, either because it is a given 
or perhaps because it is perceived 
as problematic. Yet the evidence 
suggests that the impact of ICT on 
schools, teachers and learners can 
increase the effect of other initiatives, 
for example reducing learner drop- 
out, efficiency gains, key competence 
development, improved teaching and 
school autonomy. 

Although the studies reviewed in 
STEPS provided a generally positive 
picture of ICT impact, information is 
patchy and tends to focus on inputs. 
More research is needed into the 
impact of ICT on learning outcomes, 
and in other sectors, such as second- 
ary education, and to identify transfer- 
able interventions. More international 
cooperation on regular benchmark- 
ing and lessons learned, definitions 
and methodologies would help to 
assess the return on investments in 
technology in education, and enable 
teachers, school leaders and policy- 
shapers to make sound decisions. As 
Michael Trucano of the World Bank 
recently said: 

It is necessary to have new types of 
evaluation in place and new monitor- 
ing indicators. The impact of ICTs on 
learning and future employment is still 
debatable, precisely because there is 
no standard methodology. (Trucano, 
2009 ). 
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